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GAO 


l'nit«*d  State** 

General  Acrountinti  Office 
Washington,  D.C.  2054H 


Comptroller  General 
of  the  I’nited  States 

B-221769 


April  3,  1986 

The  Honorable  William  V.  Roth,  Jr.. 

Chairman,  Comniictee  on  Governmental  Affairs 
United  States  Senate 

Dear  Mr.  Chairman: 

This  report  responds  to  your  February  26,  1985,  letter  asking  the 
General  Accounting  Office  to  evaluate  policies  and  procedures  for 
technical  risk  assessment  In  the  Department  of  Defense.  In  the  report, 
we  make  six  reconsaendatlons ,  covering  basic  risk  assessment  concepts, 
ftolicy,  and  operational  procedures.  Each  recommendation  is  directed  to 
the  Secretary  of  Defense. 

Officials  of  DOD  were  asked  to  comment  on  the  draft  of  the  report. 

Their  corancnts  appear  with  our  answers  in  appendix  111.  DOD  generally 
concurred  with  our  findings  and  recommendations ,  but  we  believe  it  is 
critical  to  monitor  DOD's  further  efforts.  For  exasple,  the  new  risk 
assessment  handbook  to  be  prepared  by  the  Defense  Systems  Management 
College  should  cover  the  assessment  of  technical  risk,  not  Just  the 
management  of  program  risk  In  general.  Coming  hearings  on  DOD 
management  will  present  an  opportunity  for  the  further  review  of 
technical  risk  assessment  In  DOD  and  for  the  direct  expression  of 
continuing  congressional  interest  in  this  subject. 

As  we  agreed  with  your  office,  unless  you  publicly  announce  the  contents 
of  this  report  earlier,  we  plan  no  further  distribution  of  it  until  30 
days  from  the  date  of  the  report.  At  that  time,  we  will  send  copies  to 
those  who  are  Interested  and  will  make  copies  available  to  others  upon 
request . 


Charles  A.  Bowsher 
Comptroller  General 
of  the  United  States 


Executive  Summary 


Technical  risks  are  inherent  in  tlu'  development  of  new  weaixm  systems 
whose  iX'rformance  requirements  exceed  the  capahililies  of  current 
weaixm  systenis.  If  not  anticipapsl  and  inanafted  ear)\'  in  the  acquisition 
pnK-ess,  these  risks  can  have  profound  «*ffects  on  a  program  s  ct»st  and 
schedule  and.  ultimately,  the  efh-ctivetu'ss  of  the  arnu'd  forces. 

The  Pejiartment  of  DcTeasc' dxx))  has  identified  technical  problems  as  a 
major  factor  in  cost  premth  and  schedule  delays  and  has  reixirttsi  that 
the  level  of  tcs  lmical  risk  dirc'C'tly  affects  dcxisions  on  fnrtlier  dec  eloo- 
ment.  In  IflHl.  ixit)  called  for  a  greater  ns«'  of  quantitative  risk  assess¬ 
ments  to  supivirt  the  bnd>li-tinR  of  extra  funds  to  cuver  tis  hnical  risk. 

Ity  IfkS.)  ixiD  informed  the C'on>lrc*ss  that  the  vrcccc-s  had  irn[iiement(xi 
this  initiaii' f.'^ 


IV“spite  the  critical  value  of  tis  hnical  risk  assessment  and  its  rejxirted 
prominc'iice  in  Iiotrs  acquisition  decisions.  ver>  little  is  known  alKiiit 
either  its  characteristics  or  the  information  on  risk  that  is  made  avail¬ 
able  to  pronram  managers  and  rco  iewers. 

The  .S-nate  Govc'rnmental  .\ffairs  ('ommittcH'  aske-d  c; ao  to  examine  ciir- 
rc'nt  ixn>  jxilicas  and  practices  novennnn  llie  asscssnumt  of  technical 
risk  and  rejKirt  on  the*  quality  and  availability  of  pop  s  tes  hnical  nsk 
information. 


Background 


I  cH  hnical  nsk  asvssmc'nt  for  a  wi‘a|xin  system  Immiiij  decelttix-d  is  the 
rcsjMinsibility  cf  the  system's  pro'tram  manattement  office  The  tiiiiixtses 
of  asMssmcnt  fjc-twrally  include  identifvi/ij;  tis  bnical  |>ri>blems  ilmi  may 
ck;  nr.  ralinn  du-  likclilwMKl  of  their  ih  cniTcnce.  and  estimatiiu;  the  extra 
funds  lus'ded  to  wive  them  The  rc'siilts  are  to  In-  uvhI  to  >;iiiile  tes  bnn  al 
(hs  isions  and  proftram  schedulintt  and  biidf;el:nt; 

To  examine  current  pot>  jxilicus  and  practices,  c;  co  obtained  relec  ant 
documents,  inti-rc  lecccd  r<-|ires<  ntaliv<‘s  of  ixiiiand  the  sc  rv  icis.  ami 
analyzcsl  risk-related  efforts  m  l’-”'  program  offices  ^  ovenu^^  ali  iua,i“r 
wc-a(Hin  systems  relevant  toiicos  pur|>oses  In  iHt  t-mUT  li'S^.  decelop 
meni  anil  |inMluction  i-osts  of  t hew  scstems  to>;ell;er  were  estimalisl  to 
excis'd  -SlSil  t>illion 
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ResulLs  in  Brief 


I 


Principal  Findings 


Risk  Assessme  nt  Guidance 


Design  Criteria 


Implementation 


iKin's  (xtivi-ni  ivith  risk  jiikI  ils  mi 

(Icrmisi'.  tKiuhas  lui  cloar  <lc>fiiini<m  of  Ua  lmii  al  risk  aiai  lia^ia-' 
o|H'<l  advico  or  (raining  snfl'iiiont  tci  gmilr  (In*  sclci  iion  and  im|i|cnicnia- 
tion  of  various  analytical  approaches  ( pp,  2  \  a  and  tiS  i'.!i  i 

In  most  of  the  2a  progrant  ol flit's  tiAo  rcvicwci).  the  design  and  imple¬ 
mentation  of  efforts  to  assi'ss  ii'ihnical  risk  ha\enot  met  minimal  stan¬ 
dards  of  ()iia!it> .  Mssential  information  on  assi-ssment  prm  ediires  and 
n*sulis  hits  often  not  Invn  available  to  program  managers  or  re\  iewei-s 
( pp. I  and  <i2-7l  i 


I«*o  has  idi-ntifii'ii  many  tifhnical  risk  atiproaehes.  Ix'th  quantitative 
and  qualiiatne  lUit  there  is  insufficient  jxilii  y  arul  training  to  guide 
pnigram  managers  in  the  si'Uvtmn  of  suitable  aiipro  u  lies  Fun  her.  uo 
standard  <Iefiiulion  of  technieal  risk  exists  within  l»*it  At  i  ttnlingK  . 
many  pmgram  offufs  have  deve|o|H'd  their  ow  n  mfitrmal  tielimiions  of 
tiH  hmeal  risk  and  risk-rating ealegoni's.  hut  i;.\o  found  them  meonsis- 
tent  and  vimetinu's eontra<luior>'.  IV'spite  imp  s  IfiSl  initiative,  nmie  of 
the  2*>  program  offer's  had  iimdueted  a  quanlitalive  te«  Imn  al  risk 
assi-ssment  to  sutijioii  Iniilgeting  for  risk  ipp.  2-1-dd.  -M-iiJ.  ainl 

t'xS-dh  I. 


IVsaiiv  iKip  had  tu>t  deM'loixaJ  standards  for  its  as"H*%sjnenls.  i;  \i) 
denvisl  entena  from  management  (inneiples  anil  pre\  nms  rescareh  mi 
risk.  Thesi-  are  pri>s|wstive  Jissi'ssment.  [>lanneil  prmvilun's.  iI<k  iimenla- 
tion.  explicit  attention  to  tis  hnical  nsk.  and  n-as'-i'ssmem  m  h  ii’jui- 
Mtion  ptiH-v  All :  >gram  offut's  had  made  some  effort  to  idenlifv 

their  lis  tuucal  risK'  h  ■  mly  H  efforts  met  tliesi’  criteria  The  remaining 
22  addri'ssisl  nsk  lu  k>  wa>  hut  did  not  fulfill  one  or  more  of  ihe 
iriten.npp 


Turning  from  di'sign  to  implementation.  i;\n  found  that  few  of  the  2'> 
program  offiei's'  nsk  efl'orts  wen-  •  arriisi  out  m  v.avs  hkelv  to  prmimi 
the  most  acnirate  and  useful  results  In  this  regard.  I  program  olfm-s 
fiad  [irov  idisl  a  Tesi  nption  of  iis  hriical  [>rohlems  and  a  rating  of  rixk 
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KxMvUtr  Stuaifury 


assossniont  rt*sulLs  and  procc'dures.  DOD  also  expressed  rehictance  lo 
place  further  rixiuirements  on  program  management  and  argued  that 
cost  growth  has  declined  to  about  1  percent,  rendering  such  require¬ 
ments  uniK“cessary  (pp.  113-21 ). 

GAO  bclievi's  that  the  findings  demonstrate  a  need  for  more  clarity  in. 
and  attention  to,  Pt  hnical  risk  assessment  in  pod.  The  findings  do  not 
suggest  that  technical  nsk  is  more  critical  than  cost  or  schedule  nsk  or 
that  MOD  S  attention  to  cost  or  schedule  rusk  can  be  reduced,  gao  lx'ii"ves 
greater  consistency  in  as-seasment  concepts  and  procedure-,  is  required 
but  also  rc*cognizes  'he  need  for  tailoring  as.sessment.s  to  par1icui«.  ,u.- 
grams.  gao  did  not  examine  effects,  but  since  most  of  txni  s  assessments 
have  not  met  minimal  standards  of  quality,  it  is  unlikely  that  they  have 
ctmtributi'd  to  any  reductions  in  cost  growth. 
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fajir.pr'.-,  i'.t 

K'-s^'art  h.  developnient.  lest,  and  evai-.ation 

Ib'moteiy  l’iloi<sl  \  «-bn  !<> 

S_\  stenis  Ai  ()msit  ion  Ites  lew  ( 'oi.ni  il 
Short  K.injte  Air  I  H  l.  iw  Command  and  Coniiol 
System 

Sbort-Kan^*'  Att-'i<  k  Missile  il 
Swbm.inne  .\d\aniisl  Comb.it  System 
T  I'  Tr.iininit  S\  stem 
Total  risk  .iss4*smii^  ntsi  i-simiate 
TridenI  II  D".  \^ea|M»n  System 


rMvti 


liSO  Pt.MI>M«STrrhnl.-al  Ki«k  S*..-*.inrnl 


wj'lir  I 


ntrodiictio'" 


i'i'<  I.  .11  .il  risk'.  .Ill'  mlHTciil  111  Itic  nf  new  \m'.i|«.i'.  sy  ii-ins, 

\\li(is<'  ;i(l\  ;in< nl  |«'rfi*rin.uii <•  ri'<,iiin  iiH'iil >  iii.iy  i-m  (t‘<)  ilic  i  .ij ..iliiiiui's 
nf  i  lirri'ill  !(•(  hn'iiii>;y  NmI  to  ;iiu  ti-i  lin;<  :il  risks  t^'lnl  i'  .ilvd 

(luring:  tlu'  <1('\ i  l'iiiiiii'iil  priK-^-ss rrr.iti's  tlir  i><i'.riit;.il  for  s<  lnsjiilini;  muf 
cos!  |irot)li  tns  ;i:hI,  w.irv,  llio  |nr..sil)iliiy  iliiil  a  systr.’n  wiil  f.ol  in  tr.n  l 
ils  (Icsiun  siKi  if  H  a'  ions  .mil  w  ill  not  fund  ion  as  iiiIi'IIiIihI  In  l,:ii'  with 
tills,  a  IMS.'!  Air  Koro'  n-iwirt  o»  an  "affor<l.il)li'  ai  <niisiiion  .i|>iiro.ii  ti  " 
found  iis  linii  .1!  prolili  ins  a  fa<  lor  in  inori'  Pli.iii  .'>n  ;K'r<'i'nl  ol  Itic  pto- 
^franis  tnat  cxiwriduotl  cost  growth 


It  is  iiiidcrstaiid.iblc  that  t(s  linical  problems  may  occur  in  Ibe  dcsciop- 
ment  of  systems  tbal  most  at  teevo  |h'.  fonn.iiu  e  jjo.ils  N-yond  ,iny  yet 
allainetl,  ;ls  f<.r  example  with  ib»'  nectl  for  si^nifu  ant  imjirov  erneiils  o, 
ttie  al  l  iiracy  of  the  siibmanne  laiim  lietl  'rrident  I!  missile  over  the  1  ri- 
(Jei'.t  I.  hill  It  is  iiriixir-.anl  to  reco^ni/.e  that  lis  bmeal  I'tohlems  rnay 
oei  nr  111  time  to  plan  am!  boditet  for  vilvim;  tlicin  and  to  sjmm  ify  (-issilile 
atiernat i\ f  Icclmii  al  apiiroaches  'ret  hnieal  risk  nss4'ssment  is  the  pio- 
ci'ss  f..r  identif.%  m);  and  t-valualiii);  the  iKilential  for  j>crfoi m.nu e 
iirotil.-nis, 

lbs  o^tm/tnn  the  tia/,.irds  of  not  'intu  ipatinn  let  hnn  al  risks,  the  I  w  p.irl 
tneiil  of  I *ef<  !'.s<‘  ( ;ii  •(>  1  tuLs  lot  usetf  t;ri  the  tnssf  to  identify  and  tor 
lechiiital  risk  III  defens<  prtalii',  lion  in  xaranis  ways 

1  ,\s  early  as  Id'.'*,  the  tlepnty  s<s  relary  of  Iv-fens*'  dins  tisi  if.e  s<s  re 
t.ines  of  the  anoed  siTv  ites  to  identify  .ire.is  of  hii:li  Its  him  al  iisk  do 
formal  risK  amil\  sis.  and  im  hide  t  xpln  1'.  loiisider.ition  ot  risk  .isse^s- 
ment,  rediict ion.  ami  avoid.ims' m  iiiana;;iii>;  weajton  systems 
aixpiisit  ion 


J  in  1  i'Hl .  t  he  depiily  St  cretary  "f  1  teleri'^’  ris-oniim  nd-.'.J  I  hat  ea'  ti  ser- 
\  ic."  e\(>ami  Its  cfloiis  to  1)11  intify  the  Its  him  ,il  n  .ks  of  systems  tiem>; 
ilex  fit  .{H'd  and  to  alk  «  ate  I  on  Is  to  deal  xxit  h  t  hese  risks  (This  ns  sun 
mem  fat  ion.  know  11  a.s  Initi.itix  e  11 ,  is  disciiss<sl  in  t  hapter  'J  1 


d  In  nsi'iit  test  imony  iH-fore  t  he  f'onvtress.  1  lefense  of  In  nils  si  ,iii  d  t  li.il 
fundmit  wit'ihl  l>e  .ipproxed  for  sy-Ieiiis  with  onix  !o\v  or  nnsl.  i.ite  le<  h 
nil  .1!  risk  hill  ideniif  \  mit  sm  h  s\  sterns  t» >sos  j)! olilcms.  snn  e  ' •  ■  has 
st.iteil  t  h.it  I  ,it  inks  ol  risk  are  s.iht's  t  ixe  ,iml  t  h.il  it  ls  ms  ess.tr \  lo  h  ' 

I  .Mlioiis  in  t  .ilekori/mk  risks  as  fn.kh.  niotler  .ite,  or  low 
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1  lie  1 « >1 1  .u  HUC'I!  ii'li  [ii  I M  t  s«.  In  t  iiir  jiii’V.  Vft  Nnmi-  I  .iiMi  !i.i!  it  ^  v.  ii  ti  | !  .!■ 
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lliulcrsl.iiul  the  issues  iti\  iil\  ed  iti  Ii  fi.Mi  ;«!  r  isk  .issessim  nl ,  Im  e.ins.. 
ntu-iitmn  In  ((■(  lim;  al  risks  is  nspnris)  m  liiese  phases  A<  e  •  ii  h's  isinn 
jxiint .  t  here  are  v\ cral  le\  els  d!  re\  lew  .  ( iilniinai  iiiji  u  it  h  '  k  1  »efi  use 
Systems  A(  (Jilisit  Hill  lieview  ('<11111(11  I  I»s.\l{i  J  er.  l(  di-ii  i;.(l(  (1  ti\  lie 
IrsMa  .  a  Syslcius  Aojiiisitinti  Keview  Coihh  iI  (s\K(  )  'Vilhin  tli<-  apjiin 
pnale  v.-\  iee  T I H •  I >Sv» la  pnivides  ;,(K  ls<ir>’  Mlpli.  ’  .  t.i  the  vi  re!ar\  nt 
iK'lense.  wild  is  the  deeidir.^;  aiithunty 


For  ea(  Ii  ntajor  w  :  ajKiti.  there  are  (otir  phas<-s  of  aediasiiinri.  (he  fii  st 
three  of  w  hii  h  end  wit  h  a  ‘'iiii  lest  one  d(s  iskhi  hy  |  he  I  is>\ia  or  dele- 
Hal(‘d  (<»  t he  VIA  ire  .<  I  lie  j. liases  are  i  mii cpI  e\ jilorat  i<  >it.  o'-innii 
sir.iti'iti  and  \  alidati.in.  full  s.  .de  de\ eU.pn'eiii .  and  pt.slui  •  iiai  ,.nd 
dephd  nifiit.  These  ph,.sj-s  an<l  their  rele\  aiu c  In  t( a  hna  al  issues  .iie 
desxTitxai  Ih'Iow  and  siiinmari/.e<l  |ti  lii^iire  1 .1 


ftgr  1  I 


(lAH  TK.MI^  6  1  n  knlrfe) 


C*!up(«^r  1 
lntr*Mi(K‘tk>n 


CluipCer  1 
IntruducUon 


Concept  Exploration 


Demonstration  and 
Validation 


Justification  for  initiatinp,  development  of  a  new  system  is  provided  by  a 
"m't'd  determinnion."  which  is  part  of  pop's  planning,  piopramming, 
and  budgeting  system  and  is  normally  submitted  when  funds  for  the 
program  objectives  memorandum  budget  year  aie  rt^quested.  One  con¬ 
sideration  in  establishing  need  is  technological  advancement.  The  si'cre- 
tary  provides  program  guidance  after  the  memorandum  review,  thus 
officially  sanctioning  the  start  of  the  new  program  and  authorizing 
acquisition  to  begin  when  funds  are  available. 

A  program  management  office  then  acquires  information  necessary  lO 
select  the  best  altemativ  es  for  system  concepts  and  the  devolopmen"  of 
hardware  and  software.  It  also  establishes  the  technical  specifications 
and  economic  basis  for  the  propost“d  system  and  develops  a  statement  of 
the  objectives,  responsibilities,  resources,  and  schedule  for  .ill  vest  and 
evaluation  efforts.  Une  program  responsibility  in  this  ph>j>o  is  to  iden¬ 
tify  cntical  technical  issues  for  subsequent  resolutior,  in  an  er.''ort  to 
minimize  future  problems. 

At  milestone  1.  the  r*“quirement  for  the  program  is  reviewed  and  vali¬ 
dated,  the  validation  being  bast'd  upon  this  preliminary  evaluation  of 
the  system  cona'pts,  cost,  schedule,  readiness  objectives,  and 
affordability.  The  milestone  1  dwision  establishes  thresholds  and  objec¬ 
tives  to  1k'  met  and  reviewed  at  milestone  11,  the  acquisition  strategy 
(including  the  nature  and  liming  of  the  next  decisivin  txiint).  and  a  not- 
to-U'-exceedod  dollar  threshold  that  will  carry  the  program  through 
milestone  11. 


Uuring  this  pha.se.  the  iirogram  management  office  accomplishes 
variety  of  Ptsks  relevant  to  the  technical  issues.  It  verifies  pielimi  iry 
design  and  engineering.  anal>  zes  tradiMiff  propvisals,  j)repares  a  fo  nal 
requirement  diK  ument,  and  validates  the  concept  for  the  aext  phasi 
full-scale  ilevclopment.  rn.toiyjK's  are  often  list'd  to  demonstrate  tlu 
feasibility  of  the  s\stem.  suhsN  stem,  or  compviru'nts,  system-spc'cific  test 
and  diagnostic  v'quipmcnt.  and  support  equipment.  I’lans  for  testing  ard 
evaluat  mg  the  system  are  uixlated.  The  program  office  also  ensures  th.  t 
the  risks  have  Ihh'iv  identifiv'd  and  are  av  ceptahle  and  that  realistifr  fall¬ 
back  alternatives  have  been  esiahlished.  I’lTformance  estimati's  are 
reviewed  for  consisiency  with  the  risks  iiu'olvi'd. 

This  phase  ('ikIs  with  milestone  11,  aiiproval  to  go  aht'ad  with  the  pro¬ 
gram.  The  lim.ngof  the  di'eision  is  flexible.  deiK'iiding  on  the  acquisition 
strategy  adopted  at  milestone  1.  At  mili'stone  II,  all  significant  risk  areas 
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Exemptions  from  the  A  niiijor  weaix)n  system  maj’  be  granted  exemptions  from  some  phases 

Acquisition  Phases  pna-ess.  For  example,  a  system  that  is  judged  not  to  rt'quire  a 

full  e<in"ept  exploration,  as  may  happen  with  a  follow-on  to  an  existing 
system,  may  skip  to  the  demonstration  and  validation  phase  or  eombine 
concept  exploration  with  demonstration  and  validation  into  a  single 
effort  prior  to  full-scale  development.  Milestone  reviews  may  also  Im' 
skiijped  or  delayed  if  there  are  no  distinct  concept  exploration  and  dem¬ 
onstration  and  validation  pha;jes  or  if  the  tirogram  has  l^f-en 
rest  met  u red. 


Technical  Kisk 
Assessment 


As  a  system  mov«'s  through  the  ai  quisiiion  c>cle,  the  program  offiev  is 
ri*si>onsi'oie  for  identifying,  monitoring,  and  solving  its  technical  prob¬ 
lems.  At  each  milestone,  reviewers  are  to  aiqiraiso  the  sourers  of  risk 
and  the  progress  of  the  program  office  in  rwhieing  risk,  .•\vcording  to 
ix)i)  policy.  thes<-  efforts  are  to  be  based  on  the  technical  risk  a.sse.ssment 
for  the  system. 


As.s<’ssment  has  many  possible  apprfiarl;<*s.  r.siially.  one  or  more  U'<  h- 
nical  experts  identify  particular  components  of  the  system  heiiig  devcl- 
ofHsl  ;md  then  di'scriln'  or  rate  the  risk  as.so<  iati<i  wiih  each  comixmcnt. 
Their  ratirgs  may  refUs  t  the  level  of  risk  and  sometimes  also  ri’riei  t  the 
consiKiueiK’cs  of  jHissible  i»,‘chnieal  problems  for  the  lost.  sclwdiile.  or 
IH’rformam  e  of  the  overall  system.  Katings  can  Ik*  expre  >.M'd  in  se\  eral 
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Methodology 


The  Questions  We 
Answered 


formats — i'xamplos  an-  a  thrt'i*-f>oint  srali-  ranniriR  from  liinli  to  mod¬ 
e-rate-  to  low  risk  and  a  jirobabilistii.  estimate-  of  tlic  ebame  tliai  tech¬ 
nical  problems  will  <K<  nr.  The-  ralinfls  can.  in  turn.  Im-  baseei  on  various 
source's  of  infe>rmat-.e»n.  such  as  exjK-rt  judgmen!,  test  eir  simulation 
n-sults,  and  jiublisiu  d  te-ehnieal/eiKirts  on  similar  systems.  Finally, 
some  iL-vsessiiieiits  e'over  emly  te-chnical  risk,  w  hil-.-  ethers  cover  te-chnicai 
alonj?  V  lib.  cosi  anei  s<  hi-elule  problems  or  e-stiniate-  the  imjilicalions  of 
te-e'lmical  p;oble-ms  f.»r  overall  proiiram  cost  anil  se-ht-efule-.  (Kai  h  of  the-se 
compeiiie-nts  eif  program  risk — technieal.  cost,  and  sclie-elule  risk — is  crit¬ 
ical.  and  e-ae  h  merits  e  areful  as.scs.sment.) 

Chajiter  2  jirovir  d:-ti  de-d  example-s  of  various  asse-ssr  le-nt 
approaihes.  Ch.-p  er  3  iele-ntifie-s  e  riteria  for  appraisin>>  the  quality  of 
risk  iLsse-ssments  an  'i  ele-scrilH-s  the  methods  pern  currently  usc-s  to 
manage  the  deve-lie'-me;;.  of  new  systems. 


KecoHnizin^  tliat  failure  to  ade-quately  a.s.se'ss  the  te-chnical  risks  fi>r  prev- 
;*ranis  can  re-siilt  in  cxee-ssive  changes  in  desijin.  prolonjte'd  delays,  and 
substantial  cost  overnins,  the  Se-nate  Covemmental  Affairs  Committi-e 
Hske-d  us  to  e-xantiiu-  l^v’.s  polieios  goveniinK  te-e-hnica*  risk  and  tei 
review  the  quality  of  l>i)i)‘s  eiirrent  assessment  preKvdure-s  and 
applications. 


To  descrilH-  ixin’s  efforts  to  identify  technieal  risks  in  the  development 
of  new  systems,  wo  feirmulate-d  six  evaluation  questiems  e'ovcring  assess- 
me-nl  i»eilieie*s  and  praetie-e-s  acreiss  the  ihrc-e-  se-iA  ices. 

1.  How  doe-s  the  De'i>artm<-nt  eif  De-fense  ile-fine  te  ehnieal  risk?  In  addi¬ 
tion  to  determinnK  how  ikio  and  the  arme-d  se'r\  ice'.s  define  tee  hnical 
risk,  wc  l(Kike*el  for  differenee-s  in  definition  or  ambigiiilic-s  in  meaning 
that  might  affe-ct  the-  way  assc-ssmenls  are  porforme-d. 

2.  'That  guielaiue- d<>e-s  ixip  proy ielo  fq^r  a'^t-sMng  ti'chnind  risk '  Ik-eaiiso 
defe-nse-  .systi-m  eJeve-lopnii-nt  is  unique,  we  wanle-rl  to  le-am  what  a.s,s(-ss- 
ment  afiiiroae  he-s,  if  any.  hod  has  elevelojK-d  or  promoted  for  the  use  of 
the  jiri,g'‘am  maiiageme-m  eiffices. 

3.  Lkneiliaye- the- soia  iers  iinjijenu-np'd  hiituiliv^  1  r.’  'W  sought  to  deter¬ 
mine-  w  hi-the-r  s[K-cific  jKilie-ies  on  (e-chnical  risk  assessment  have 
ri-siilte-d  from  Initiative-  1 1.  the  11*81  i»>d  ri-edinmeruiaticn  to  the  :'eer\  ices 
for  quantifying  and  hudge-ting  for  te-ehnieal  risk.  We-  aiso  sought  to 
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examine  aifferences  in  the  way  ihe  service's  approach  technical  i  isk 
assessment 

4.  What  arc  the  characteristics  of  current  efforts  to  identify  the  tech¬ 
nical  risks  of  new’  systems?  This  question,  aimed  at  describin>;  efforts  to 
identify  U'chnical  risks  for  systerres  now  under  deyelopmcnt,  encom¬ 
passed  the  largest  set  of  issues.  It  included,  for  example,  determining 
when  these  efforts  are  performed  and  whether  they  arc  being 
documented. 

5.  How  are  efforts  to  identify  technical  risks  implemented?  We  sought  to 
learn  what  formats  arc  u.sed  to  rate  risk,  whether  the  ratings  co^  er  s[H‘- 
cific  subsystems  or  only  a  system  as  a  whole,  and  how  data  on  tt*chnica' 
risk  are  collected. 

C-  What  information  on  t(*chnical  risk  is  ayailable  to  decisionmakers  in 
the  reyiew  pr<K-ess?  This  question  completed  our  examination  of  tlie 
acquisition  process  and,  together  with  the  fiyc  other  evaluation  que.s- 
tions,  provided  a  framework  for  examining  pod’s  technical  risk  as.v'ss- 
ment  policies,  procedures,  and  applications. 


To  answer  the  evaluation  questions,  we  collected  info; mai  :'>n  from  the 
Office  of  the  St'cretary  of  EHTeivse,  service  headquarters  staff,  schinils, 
laboratories,  and  defense  contractors.  Our  principal  data  collis  tion 
effort  was  gathering  extensive  information  on  technical  risk  ;lss<*s.s- 
ments  from  25  program  offici'S  managing  the  develojiment  of  new  sys¬ 
tems.  To  obtain  a  full  understanding  of  technical  risk  ass**ssment 
throughout  non.  we  examined  all  three  .services  (the  Army,  .Navy,  and 
Air  Force)  and  the  differences  between  them. 

We  defined  our  universe  of  systems  as  all  "major  acquisitions"  going 
through  Ps\KC  review.  Major  acquisitions  are  m()rc  costly,  jxisi-  greater 
risks  in  development,  and  are  more  intensively  reviewed  hy  ihi'  .m'(  rc- 
tary  and  the  Congrt'ss  than  other  acquisitions.  Therefore,  we  saw  them 
as  the  most  likely  to  have  had  acquisition  impn.vcmcnl  initiatiws  and 
many  relatc*d  program  management  functions  implcmc-nted.  \Sc 
exclud(*d  from  our  study  some  of  the  4.'1  major  acquiMtions  (hat  were 
under  dc'Velopment  cm  .July  31.  1981,  for  tliree  reasons 

1 .  Fnigrams  very  early  in  the  acquisition  cycle  larked  the  d(K  umenta- 
tion  we  needed  and  had  not  progressed  through  the  rc\  iew  pi(«  -  ss  Pnt- 
grams  very  late  in  the  cycle,  those  already  in  production,  had  already 
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passed  through  the  review  process,  in  which  it  had  been  certifii'd  tliat 
all  technical  risks  had  bei>n  n^soK  ed.  In  other  words,  we  excluded  pro- 
graeis  that  had  n(»t  yet  passed  milestone  I  and  those  that  had  already 
pas;  “d  milestone  111.  The  programs  we  examined  were  in  cither  the  dem¬ 
on  .tr  iti  on  and  validation  phase  or  full-scale  development. 

■;  !xcludc*d  ship  hull  programs  (but  not  .ship  systems  such  as  elec¬ 
tronics)  because  of  the  long  periods  of  time  (up  to  10  years)  it  takes  to 
build  them  and  the  generally  low  level  of  technicaJ  risk  associated  with 
them. 

3.  Because  of  Don’s  administrative  decisions,  we  excluded  the  Army's 
guided  antimortar  projectile,  non  cancfllod  the  program  b«.'fore  we  were 
able  to  collect  data  from  the  program  management  office.  And  we 
excluded  the  Navy’s  tactical  microwave  landing  system,  which  non 
included  among  its  major  acquisitions  to  ensure  that  the  sec  retary  would 
review  one  of  the  system’s  conuxments  but  exempted  from  ds.'wHC  mile- 
sUme  reviews  (and,  hence,  it  fell  outside  our  parameters). 

This  left  25  systems  in  our  universe,  including  5  Army,  1 1  Navy,  and  9 
Air  Force  systems.  (We  classified  joint-service  programs  aco»rding  to  the 
scn'ice  with  lead  responsibility  for  development.)  In  December  1984.  the 
projt'cted  development  luid  production  wsts  of  these  progrimts  exceeded 
$18(1  billion.  They  are  di'scribt'd  briefly  in  apix.*ndix  1  and  listed  with 
their  stagt's  of  development  in  table  1.1. 
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Table  1.1;  The  25  Major  Syatema  We 
Examined  and  Their  Miletlono  Review 
Status  on  September  IS,  1984 


The  Approach  We  Used  to 
Collect  and  Analyze  Data 


Syttem 

Abbreviation 

Ser  /ice 

Between  milestonet  1  end  II 

AnliiutSTianne  Wa.  .'a/e  Slandotl  Weapon 

^  'asw  sov/ 

Navy 

Advanced  Taclical  Radar  System 

ATRS 

N.ivy 

C17A  Aiflilt  Airc/a!!  System 

C  17A 

Alt  Force 

CV  Innerzone  AntisuDmat/r/e  Warfare  Heiicoplet  • 

CVHELO 

Navy 

High  Freqi.)ency  Anti  Jammer 

HFAJ 

Navy 

Inter  Service/ Aaency  Aulomalod  Message  Processing 
Exchange 

1  S/A  AMPE 

Air  Force 

Joint  Surveillance  and  Target  Attack  Radar  System 

jstars 

Air  Force 

Mark  XV  Identification  Friend  or  Foe 

Mark  XV  IFF 

Air  Force 

Multiple  Launch  Rocket  System/Terminai  Guidance  Warhead 

mlrsagw 

Army 

Short  Range  Air  Defense  Command  and  Control  System 

SHORAD  C2 

Army 

Short  Range  Attack  Missile  It 

SRAM  It 

Air  Force 

T-45  Training  System 

T45TS 

Navy 

V  22  Osprey 

V-22  Osprey 

Navy 

B«tw««n  milettorMt  II  arKl  111 

Army  Helicopter  Improvement  Program 

AHIP 

Army 

Advanced  Lightweight  Torpedo 

ALWT 

Navy 

Advanced  Medium  Range  Airto-Aff  k'rssrie 

AMRAAM 

Air  Force 

Antisateiiite  Weapon 

ASAT 

A:r  Force 

Airtxxne  Seif  Protection  Jammer 

ASPJ 

Navy 

Joint  Tactical  information  Distribution  System 

JT  IDS  (Air 

Force) 

Air  Force 

Joint  Tactical  Inlormation  Distribution  System 

JTIDS 

(Navy) 

Navy 

Ml  Abrams  Tank  Enhancemeni 

M1A1 

Army 

NAVSTAR  Global  Positioning  System  User  Equipment 

NAVSTAR 

User 

Equipment 

Air  Force 

Remotely  Piloted  Vehicle 

RPV 

Army 

Submarir>e  Advanced  Combat  Systerri 

S'JBACS 

Navy 

Trident  II  D5  Weapon  System 

T  rident  II 
(DS) 

Navy 

To  iin.swor  the  six  evaluation  questions,  we  obtaintxl  tlcK  tiinents  to  pro¬ 
vide  evidence  of  service  policies  and  pronrant  nianajtenieiit  activities 
and  a)nduct<‘d  stnicturerl  inler\iews  loensiin*  that  information  w;ls 
consistently  obtained  from  the  program  manajtement  offices.  (Jiir  data 
sources  are  di.scus.v-d  briefly  lielow  and  more  fully  in  chapters  2  and  3 
Table  1.2  Kivc-s  an  outline  of  the  p.iinary  data  vnira-s  by  evaluation 
question. 
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Table  1.2:  The  Primary  Data  Sourcaa  lor  Our  Evaluation  Ouestiona 

Publicationa, 
Office  of  the 

Secretary  of 
Defanaa,  and 

Program 

service 

management 

Evaluation  queatlon 

heaiquartart 

School  office 

Lab 

Contractor 

1  How  does  DOO  Oeli.-ie  ternmcal  risl^’’ 

X 

2  What  guidancr  does  (XiD  provide  !or  assessing  technical  nsk"’ 

X 

X 

3  How  have  the  services  implemented  Initiative  IP 

X 

4  What  are  thie  characteristics  of  cunent  efforts  to  -dentify  the  technical  risks 
of  new  systems'’ 

X 

X 

X 

5  How  are  efforts  to  idcniily  technical  risk  implemented'’ 

X 

X 

X 

6  What  information  on  technical  risk  is  available  to  decisionmakers  in  the 
review  process'’ 

X 

For  qucsti(»n  1,  on  ton’s  definition  of  risk,  we  gathered  publications  that 
define  tc'chnical  risk,  including,  reguIaM(>ns  and  <^*her  documents  specifi¬ 
cally  about  risk  assessment  for  non  and  the  three  serv  in's. 

F'or  quv.stion  2.  on  ton  guidance,  to  gain  background  information  on  the 
approaches  to  technical  risk  assessment  available  within  non.  wo  list'd 
documents  and  interviews  at  the  Offia'  of  the  Secretary  of  iK'fense  and 
the  Defense  Systems  Management  College,  the  Amiy  Uigistics  Manage¬ 
ment  Center,  the  Naval  Postgraduate  Sch<x>l.  and  the  Air  Force  Institute 
of  Technologj'. 

For  question  3.  on  Initiative  1 1.  our  primary’  sources  were  dcxrumcnts 
(regulations,  memoranda,  and  ixilicy  statements  that  repri'sented  offi¬ 
cial  res[x.n.ses  to  Initiative  1 1 )  juid  interv  iews  w  ith  staff  in  the  Office  of 
the  Secretary  of  Defen.se  and  with  individuals  at  the  headquarters  of  the 
thrtv  sers  icc's  who  were  involved  in  decisions  relevant  to  the  initiative. 

Far  questions  4,  5,  and  0,  on  risk  effort  characteristics,  i.mplemimtatioji, 
and  information  for  deci.sionmakers.  the  primary  data  source  was  an  in- 
depih  census  of  our  universe  of  programs.  We  gathered  documents  anri 
interv  iew  information  from  program  management  offict's  on  the  risk- 
identification  efforts  jK'rfonned  for  major  systems  under  develoiimeni 
in  the  Army,  Navy,  and  Air  Force.  (We  al.s<i  conducted  exploratory  inter¬ 
views  with  individuals  in  HOD  and  at  the  headquarters  of  each  si'rvice.) 
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iiu  risk  asM'ssmmts  iHTfnniu-il  fur  tlir  \vr;t|><Mi 
sysloms  jiinJ  <i<M  iimi-nt.s  ;iv;ii);ihlc  for  tlu-  iis.\KC  riA  u'W.  Amonji  tlic  (1(k  u- 
monts  niiuin'd  by  tin-  lr<\K«'  won-  systmi  loiui'tit  piiin-rs.  di-i  ishmi  iiKir- 
dinatinn  paix-rs.  intr^'.iatixl  pninrain  sumniarics,  U'st  and  rvaluatinn 
master  plans.  arqniMtiim  stratcjia-s.  and  briefing;  materials  prepared  fur 
tlio  l«s.\KC  and  the  mt\  ia‘s.  At  »-aeh  prufjram  offiie.  we  inter»  lewiKi  the 
pntjjram  niananer  and  de(m(y.  eontrat  t  nffic  ct.  chief  en^tineer.  and 
n'lu'rs,  if  any  invnlvetl  in  |H-rf<>rinin^  risk  assessment  efforts.  We  also 
tnter\  iewed  staff  at  sera  ice  lalHiratories  and  eontra'  tors  if  they  (ht- 
fnrmed  assi-ssnients  fnr  the  jiroftram  manajtemeni  office,  bin  we  did  not 
MX'k  inforin.'tiiMi  from  thesr*  soiir<t“s  iink'ss  the  pro^tram  in  inaftenu-nt 
informed  us  of  their  outside  <-ontnbuiion. 

For  hell)  in  answering  the  last  thnn*  cv.tluation  qiU'stions.  we  abai  devel- 
oiH'd  stni( Hired  interviews  wh.en  rlata  collection  across  multiple  siti-s 
was  rtxiiiired.  We  tivsi  separate-  <lata  (ulliition  instntments  for  the  pro¬ 
gram  offi<  I's.  schools.  lalMiratories.  and  contractors.  We  develo|)<‘d  a  pri¬ 
mary  inteia  iew  for  iirogram  managers,  deputy  program  m;  nagers.  <  hief 
enguusTs.  and  other  program  staff  and  an  additional  set  of  questions, 
which  we  u.setl  in  conjunt  tion  wnti  the  main  interview,  for  iiersons  who 
actually  cofiducf<-d  risk- identification  efforts.  We  iis<'d  ;  eoarate  inter¬ 
view  fortns  for  contract  officers  anti  for  program  offices  in  the  .\rmy 
that  employed  the  total  risk  ass<*ssing  cost  estimate  approach.  Forms 
were  pretesttv)  at  •>  (irogram  t)ffic<>s  during  the  planning  phase  of  this 
study.  Fuilher  information  on  the  data  lollis  tion  instruments  is  a\  ail- 
able  from  (i.\o's  I’rogram  Kvaliiation  and  .MctluMlology  Division. 

We  selis  tisi  tpialitative  tlala  analysis,  im  hiding  a  tabulation  of  \  ariables 
drawn  largel.s'  from  our  interviews  in  the  program  management  ofhees, 
as  the  approai  h  l>est  siuteil  to  the  information  we  gathered.  We  also 
analyzed  the  diK  uments  w»'  eollis  ted  in  order  to  deM nU-  the  tet  hnii  al 
risk  information  thev  lontaimsl. 

For  a  h'vv  weapon  s\  stems,  the  program  management  offices  |H-rformed 
tnti  or  more  risk  el  foils  For  these,  an  effort  was  considered  |irimarv  if 
it  was  the  one  most  frequently  mentioned  hv  res|Min(ieiits  or  was  ihe  one 
that  liad  been  tiiost  recent iy  i-onducted  or  met  more  technical  risk 
assessment  criteria  than  other  efloils  t  s<>e  chajiter  d  i  .ApiKsidix  I  men 
lions  a  variet.v  of  lei  hmcal  risk  «-valiia1ioiis  that  we  did  no!  include  in 
our  analyses  (Our  rev  lew  was  conducted  m  accordance  with  iti'iierall.v 
aci  epied  audit  ing  standards.  i 
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its  ;)iir]Kisc  and  d(‘si>;n.  our  study  liit-i  strc'uttllis  and  lunitations 
that  sl'.ould  1k‘  r»H-o>;iu/.i'd.  <>iu‘  luuitation  is  tiMt  tin-  study's  arcurary 
and  i-omplctcni'ss  of  data  (lf|H*nd  largely  tut  the  resjxjndents.  \S'tiene\  er 
[Hissihle,  infoniiation  front  one  resjxtndi'nt  was  confiniii'd.  and  inconsis¬ 
tencies  resolvtsl.  by  clieckinn  with  other  resixmdents,  inc  luiiinn  forinci 
memlH-rsof  the  projfrarn  mananenteitt  staff,  and  by  relernnn  tooffictal 
protjrant  diwuntents.  In  some  ii.stanees.  however,  the  stna  tore  of  ti  e 
program  office  or  the  natur(>  of  the  risk  effort  made  it  imiiossible  to 
obtain  furtfier  information;  thus,  for  a  few  questions  in  the  rejKirt.  some 
data  are  mis.sinn. 

A  sr“cond  limitation  derives  fnim  the  parameters  set  by  tii.i'  evaluation 
questions.  The  purivise  of  tins  study  was  to  discriminate  risk  efforts  on 
the  basis  of  lear  differenc-es  in  th.*ir  desinn  and  implementation  Our 
pariKise  was  not  to  determim*  whethi-r  the  efforts  were  aetu,ill>’  used  in 
firojjram  d  ‘cisionmakin^  or  to  compare  ihe  effettiveness  of  efforts  that 
do  and  do  not  meet  various  a.s.sessment  criteria.  Aceordinuly.  we  diil  not 
attempt  to  link  “fforts  to  outcorra's  siu  h  as  rest;  lU  turinj;  programs  or 
rediK  inn  nt'u'Oh. 

A  third  liniitation  .dsoderiir's  from  i>ur  pijr;x).s<“.  We  e.xaminerJ  only  the 
pnnessof  addressinn  Its  hnical  risk  in  weaixtn  systems  development.  W«‘ 
made  no  attemiit  to  estimate  actiud  lisk  or  the  ;iccuracy  of  statements 
alxitit  risk  for  Ihe  systems.  judnments  were  n.ade  alxiut  whu  h  sys¬ 
tems  have  hinh  risks  or  alxiut  whether  risks  should  Ik*  an  imiK*<limenl  to 
apjtrovmn  the  r-onlmuamcof  systems. 

The  stuoy  has  n<iti’wotihy  strennths  as  well  First,  our  interx  iews  were 
with  respoiuK'uts  who  have*  a  romjtn  hensive  ranUe  of  interests  and 
ex|H*rieni  es  relev  ant  to  this  topic,  meltidinn  pronram  mananei’s.  mile¬ 
stone  reviewers  in  conimand  offices  and  in  th<*  Office  of  the  ,'s<‘<-relar.v  of 
Ih'fense.  and  staff  inemlH-rs  m  pronr;».m  offices.  lalxirafories.  and  i  on- 
tractors.  We  aUo  inlerviewiil  repres<*nlalives  of  .top  scIukiIs  offermn 
coiirsi’s  on  ri'-k  In  combimition.  our  inlerv  lews  included  resjvondents 
w  ho  (ilan.  [K*!  form,  interpret.  ;md  rev  lew  nsk  efforts  and  lesjKindenls 
vv  ho  jirovide  relev  ant  traininn 

S’cond.  w  ith  t he  I'M  eptions  alread.v  noted,  we  covered  all  major  acvjuisi- 
tions  now  in  dev  elopment  Since  thesi*  riseive  Inip's  clos«-st  si  nitinv .  we 
e\|K’c!ed  risk  efforls  for  tlie-.e  systems  to  i>e  amon^t  Imli's  most  carclul 
attempts  to  identify  and  plan  for  technical  iirotileins 
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'I'lxtci.  workmH  fri>iu  s«uircfS  on  ri>k  assossini-iit  and  pto^irani 

inana^cincnl  ( list»‘<l  m  a[i|K‘ndi\  11 ),  uo  <lcvcli»|H'd  >;ciicric  cniiTia  lor 
till*  qn.iiil y  td  risk  oil'oris  T.)  oiir  knowUnl^’i*.  no  othor  m'I  of 
t  riloria  likn  tlmv  oxisls.  WhtU- oi:r  sot  is  not  nii-cssaril\' dcfinitu'c,  il 
rlin-s  offer  a  nu'aninjtlul  way  to  <liM  rirninate  risk  efforts  and  a  basis  for 
further  refining  the  c  nteria. 

Finall;,'  this  reixirt  pntvhles  new  and  imjxirtant  infonuation  I’nw  ions 
stiidii-.  ha\-e  not  sysiemalK  ally  deMTilxfl  the  i  haraelerisi ii s  of  l»>i»  s 
risk  efforts  or  the  information  thes«*  efforts  provide  to  decisionmakers 
(Ss*,  ‘or  example.  .Army  fx-parlnient.  ..rui  U  dhams  and  Anevla. 
IDHIJ  ‘  t  (hirs  d<H‘s.  providinn  a  basis  lor  evaluating  jx>'.sible  revisions  in 
relev  ;.c  iKHi  (Kiticies  and  practice's  and  for  plannin>t  studies  of  the 
effc:.'  -  r  risk  asvssme-nt  on  pniftrani  cost.s  and  se  hediiles  .ApjHTidix  III 
ceintains  comnie-nts  pot*  made  on  a  draft  of  this  rejxirt  and  oiir  resixnise 
to  tlu>  comments. 
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III  ttll^  ( liiipicr.  •At’  ;ms\MT  thf  lirst  tlirtf  fv  .tliMiiiui 

1  ilil\\  (liH‘S  Imll  (li  flltf  t.-t  IlDli  .ll  risk" 

2  tliK's  uki  prox  itif  for  tifliMu;il  risk  ” 

;)  1  low  ti,i\c  till- st-rvuvs  iiiiplcnu-nlftl  li\ili;iti\ f  11'.’ 

* 

Ciirri'iii  lx  III  (Kiln  H-s  I  ;ill  for  iho  ;ii  kiiovv  l■t^^'.lnt•ll^  anil  tlis<  ns-^ion  of  o 
syMcm's  (It  linit  .il  risks  ihroiii^liotit  llif  at  tiir.sit mn  t  yt  Ic  |{l•s[Klllslllllll y 
for  ass<-:ii|plmj;  am!  [irov  itlmj;  its  litiical  risk  tntoi ni.il ion  is  (ilat  oil  on  ihr 
|iro(;rani  niaiMvIonii-nl  oil  iff  .As  a  rfsiilt.  n-v  ifwi-rs  of  tlic  s\sifiii  may 
tia\f  (Ilf  atlv  aniakf  "f  ‘•lilaiiiiiij'.  iiilorni.il ion  Ironi  llif  [ktsiihs  w  li.i  ari' 
llif  most  fXiK-rifiu  itl  v.iili  ll  <  (Ilf  tlis.uK  aiil.ikf  is  that  ihf  inour.im  stall 
may  not  pro\  nlc  an  ohifft.vo  or  imlfiK'iitlfii:  liKik  at  llif  systmi. 
allhoiikli  llif  rf\  ifwfrs  nffd  foni|i|ftf  anil  (innjirfhfnsivf  loi  hnu  al  risk 
information  lo  m.ikf  ihfirown  i-valiialioiis. 

riif  firsi  (liK  imifiil  rfijiiiri-il  for  a(>|iri-viniJ  llif  ai  (jiiisition  of  a  ssraiHin 
sysifiii.  ".Ui'iilifaiion  for  M.iior  S\stfm  Nfu  Sl.irt.  '  niiisi  dis.  iiss  ihf 
m,iiur)!>  ol  Mf  systfiii's  lfi  hmilojty  wiili  "j'art n  iilar  fm|  h,isis  on 
ifmainink  aii'.isoi  risk  ’’  i-iliT.  at  i-ai  h  milis,tonf.  dis  isions  m.idf  al 
hitthfi  I  onim.'iid  |f\  rls  must  i.ikf  loi  tiim  al  risk  into  toiisido i  .it  ion  This 
is  lo  Im'  diH  lllllflltfll  ils  tolloWS 

1  .\l  nnlfstonc  I ,  a  s\ sifin  (  mil  fpi  pajK-i  imisi  idfiildv  Sf\  aif.iso! 

Id  him  .ll  risk .  ll  I  III  h  .Ilf  lo  l)f  If  dm  fd  i )  i  oo/h  r  fsiaii  )i  and  df  \  f  I  a  > 
llif  III .  I  Ilf  I  fdm  I  ion  lo  Ik-  \  alidaO-d  I  hrointn  Ifsi  ini!  .aid  f  \  .ilii.il  a  ni 
Ik- loro  milfsidtif  II 

2  for  imlfsioii-’s  11  ami  ill  .i  dd  isimi  iimrilin.iiii'i;  i>.i|k-i  niiisi  i  ont.nii  a 
(!is<  iissKin  ol  I  t.c  I  oil' Minini!  |fi  linii  al  i  ;sk s  o!  Ihf  s  \  si rm  1  i ir  inilfsi i jin- 
II.  I  Ills  I  •.i(»'i'  iiiiisi  .ils/i  disK  i|ss  1 1 -si  and  f  V  ahi.il  ion  m-siiIi  s  .md  shoii 

I  lial  all  snind  n  .ml  i  isk  .ii  f  as  ha\  i-  Ik-i  ii  ’■i  sop  od  and  I  h  it  1  hf  Oa  h 
n'•!ll^\  1  f< III  os  oii|\  •■nvtiiid  !  nivt  (  not  f  \  |»i  inif  lit  .il  l  fl  hf  1  s 

I  f  ol  f .!(  1 1  nn  If 'I  off  If  \  If  u  a  Ost  and  f  i  a  |o  imi  masOi  pi.  in  imisl 
il-  si  1  i1k-  i  I  ll  i.  .ll  issm  s  lo  I-.  ,i,f.  .  s-^  d  li\  Ifsi  im;  im  1  iitin,;  issm  ^ 

.It  ismk  1  roiii  Id  liiiii  al  risk 

I  1 1  all  ol  p.i:  I  ot  1  hf  s  \  -.1  f  ill  s  Id  liiiolm;  \  I  ..is  m  it  ti.  f  n  df  iiioiisi  i  ,iti  <1 
I  la  n  loi  f  ,11  1 1  III  ill  si  off  1  f  \  ll  a  an  inlf  iti  .i!  fil  1  f  0^1  .im  sn  mm,ir\  must 
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i(li'iitif>'  th<‘  tfriiiui  .il  rivks  iiiul  it!  liv  iiics  >h;il  li.ivr  Im  cn  jii,inni'<l  Ins 
•.‘iliinitji  ll.rm 

r>  K(iri‘;n  t\  \  lew.  a  paixT  t  ailed  "ai  (juimI  mii  s!ra(e>'.>'"  (iiiist 

•'UHiinan/.o  llie  let  lini'  al  risks  and  tin*  plans  U»  nsiine  nr  elinuM.iie  them 

Fill! her  vvideiv^-  •'!  <'  iP '■>  eoneerti  w  iili  its  hraeal  risk  api-sirs  ni  lesi  i- 
rreiny  iH'fore  llie  ''tninn-ss  In  hearings  Ijefore  the  llntise  Hiui^iet  (  iiin- 
•“ lips’ nil  i-ehniary  2'>.  litHd.  the  i.ndersts  retary  nf  I  K-fenst- foi  i-art  h 
ami  eii>;iiusTmk  statisi  tiial  imp  w;ls  'inakm^t  realistie  iLssessiiienis  nl 
te<  hail  a'  and  v  hisliile  risks  and  liniitin>j  f.s  hiin|nj,'i(  al  arh  ain  einenls  [n 
Ilf  im  nritnraPti  in  ntir  systems  '  ( I  '.S.  (‘nnvtress.  1  PSith.  p  .'■>nK )  ( in  Fet>- 
niar>'  J7,  IP.Hl,  in  I.eariniJs  liefnre  liie  llmiv  Arimsl  S'rvii es  Cnin- 
initiisv  till'  iindervs  retary  slatisl  tliat  Inllnw  ini;  Initiativ  e  1 1 . 

"si);nil lean!  pmjiress'  had  Ixsai  inadetnwaid  n-diH  tnii  enst  i;rnv\th 
stenunifU  frnni  tis  linn  al  risk  i  itinU  an  effnrl  that  't|iiantifies  ihe  i  nst 
reijiivtsi  tn  nvereni.  e  deNelnpiiieiit  risk  and  pmurani  llie  lipTii-K 
1  •sear*  !i.  de'-  elnpment.  lest,  and  ev  aliiatinn)  Iiinds  mssksl"  ( I' s  (  mi- 
Hress.  l'lH-1.  p  Till. 

Fin.il'.V'.  as  \\e  diseiisstsl  in  i  lia|>ter  1 .  Ini’iative  1 1  eallisi  iijxin  the  set  ■ 
Mies  tn  iiiiprnv  e  t  liei;  tts  linieal  risk  assessments  and  In  tiiiditet  fnr  leeli- 
im  al  risk.  As  a  resid. .  analysts  msnle  and  niitsnle  imi.  liave  develnix-d  up 
identifietl  apprn;.<  hes  inr  assessing  te<  hnii  al  risk 


I  til  e  apiM’ars  In  lx-  nn  siamlard  deluatinn  nf  ie<  hnn  al  risk  in  s  iIih  • 
iiinenis  and  re^ulaimns  nr  ihnse  nj  i.Pe  vr\  lies  In  s^mie  insi.iiii  es,  ihe 
let  III  "risk  "  IS  list'd  In  refer  tn  pin;tr.irn  risk  in  ilener.il  In  ni  her 
I  nst  anil's,  the  term  relers  to  one  nr  .nn  •!  her  i  nmiMineni  nf  provii  .nn  i  isk , 
sin  h  as  enst  tfroath.  m  hetliile  del.iys.  .nid  |»'rfnrman(  e  prnhlerT.s  Ifisk 
assessrnei  t  .ipprn.n  Pes  often  Ineak  (irnvtr.im  risk  into  llies<'  <  ..r  innenis 
Seine  ai'(i;  n.n  hi-s  de.il  e\eliisivel\  \MI  li  i  ine  i  mill « iiieni .  n|  hers  im  m  jx»- 
tale  tin  ire  than  one  h  ill  illl  l  lie  same  IiiiKlel  While  eai  h  nf  I  hesj'  risk  i  nrii 
)>nnent  s  |s  l  i  lt  leal  to  pr  'ttram  sin  (  ess  and  nstiiii  es  espln  it  alieni  mn. 
they  ale  lint  indeJK'lident  .\s  we  disenssisl  in  I  h.ipter  1  ,  1 1 -I  hnn  a  I  pi  np 
ll•:ns  are  appai  eii'  i\  a  in  a|nr  f.n  tm  in  ;hi  i  nst  n\  i  r'lins  in  weal  .mi  s>  s 
1'  II,>  aeijllisil  inn  l  !ieie|nre.  leelltili  al  risk  is  lelatisl  In  enst  risk  .Hill,  in 
the  s.inie  wav.  I  n  l>e.1ii|c  risk 

Ihe  I  lelellse  Sv  vieliis  .Maliar'emenl  (  <  illee,  •  ()i  I  m.  risk  as  ihe 

pi  nl  ah|t|l  V  ,11  id  1  nnsi'ijin  III  e  nt  r.'  ■!  .!■  hii  v  ini.;  s4  iii.e  del  II II si  (irn^raiil 
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— suc  h  as  idsT .  sc  hcctiili'  or  Icc  liiiical  ih'i  ronnan'  c' "  ( I  )flcMS(‘  S\  s- 
lonis.  IhK.J.  p  :ii.  Tins  di'liiiitioii  sn>:;:osts  tiiat  ratiuj^s  ol  Ic  i  hmc  a!  risk 
should  lake*  into  a>  c'ount  hoih  ilu‘  hkclilux;;'!  and  the  consi'ciiirnccs  ol 
prohloins.  Accdrdinjjly.  a  piohlorn  c  (msi'icrcd  \cr\'  iinl'kolc  niij'hi  he 
rated  '■hinh-risk  ’  heeause  ii  n  were  to  .K  c  iir,  its  eonseciuenres  tor  pro- 
>;rain  cost  or  schedule  would  he*  sevc-re.  C’oinhimnjl  prohahility  with  lon- 
setjuenccs  in  a  smule  rating ohsetires  the  nature  am'  'evel  ol  risk  Ironi 
tc'chnieal  jirohteins.  In  an>’  ease,  this  detinition  is  not  hindinn  or  even 
aeticcly  prontuljiatc'd  within  Utii. 

I«  iti's  reflulations  on  milestone  documents  do  not  provide  a  definition  of 
technical  or  program  risk,  nor  diH's  ikH'  s  directive  for  managing  risk  in 
the  transition  from  develoimient  to  [iroduction  (disc  usst'd  in  the  nevt 
sc-ction  below).  'I’he  onh  ser\  ice  regulation  we  found  with  a  definition  of 
program  risk  is  Air  Force  Ih'gnlation  Td-lo.  It  governs  source  sc-lection 
jxilicy  and  proceciiires  and  dcdines  high.  miKleratc,  and  low  risk,  slight l.\' 
parajihrased  as  follows; 

1 .  liighjisk  is  likely  to  cause  significant,  serto-js  disruption  t.t  schedule, 
increase  in  cost,  or  d<*gradation  in  iH'rforiiiaiice.  e\  t‘U  with  special  atten¬ 
tion  from  the  c-i'iitractor  and  close  governnieiit  monitoring 

2  MederaU'  iis,k  can  cause'  some  disruption  in  s.  hediile.  iiu  reasc'  in  cost, 
or  degradation  in  perforiiiam  e.  hut  siH'cial  attention  from  the  contractor 
and  close  government  nionitoring  tan  prohahlc'  owreome  the 

f't'l  K  Illtil'S. 

‘i.  I>>w  risk  has  little  (xUenijal  Idr  causing  disruption  in  sc  hi'diih  . 

UK  Tease  in  cost,  or  degradation  in  |>ertdrmanc( ;  normal  ctlort  Iron;  the 
contractor  and  normal  government  monitoring  can  jirohablv  ovc-rc  iinc' 
the  ditficultic's. 


lake  I  he  del  11  lit  ion  ot  risk  gi\  eii  b>  the  I  >e  tense  S\  steins  .Maiiagerneiit 
(dllc'ge.  the  Air  Force  doliiiitions  of  risk  levels  comhine  the  hkeliluxul 
t  hat  a  problem  will  o-,  c  iir  w  it  li  i  he  seriousness  of  its  coiiscquoiK  cs 
Moreovi'r.  the  .\ir  Force  delinitions  do  not  r.tpiire  lailmgs  of  technical 
risk  distinc  t  from  ratings  of  <  ,ist  and  sc  iiediii"  risks;  they  coinbine  tbc'sc' 
•  miiponenls  into  an  over;di  rating  of  progiatm  risk 


Air  Fctrce  Regulation  Til-lo  also  reciuires  contiaictors  to  idcntilv  risks  in 
their  projKisals  The  regiikition  siiggc'sis  that  the  program  m.inagc'iicrit 
ot  fice  should  give  the-  source  selec non  evalu.ition  hoard  th.it  rec  I'ues  the 
Iiroposalsan  inde|H-uc  ct  assessmeiii  ot  tlic  ri^ks  in  advaiuc  Ilowi'M-r 
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it  (Ijk's  not  siH’dl'y  )io\v  to  nssi’.ss  tlic  risks,  Tl>i‘  Ariv.y  and  Navy  lia\c  no 
forrvspoiulinn  rc>;iilatioiis  (Iflininji  risk. 

lias  Initiativi"  1  i  iini)oscd  iiikhi  uhi  a  standard  di-l'in..  .n  <>(  risk  ’  Siiu c 
"ttHlinolojtual  risk"  apjx'ars  in  its  title.  ■'Incc'rixirate  :lie  (  se  of  Hnd- 
jteted  Funds  .‘or  Te<  hni-lojtical  Kisk."  liuti.itive  1 1  elearl.\'  reteis  to  lecli- 
nieal  risk,  not  seluslule  or  cost  risk.  Two  years  afti'r  lnitiati\e  1 1  was 
issued,  the  deputy  s«‘eretary  ol  Defi'use  reiterated  this  |x)ini.  saying  that 
the  services  had  irt|)leiner.ted  procedures  to  tiudjiet  for  "'tei  hnolotlical 
risk"  (t'.S.  Congress.  ItrH.'hi.  j>p.  2.'>2.  27(t.  and  2S  l  i.  Vet  Tiiv  K.  the  total 
risk'  assessinjl  cost  estimate  method  rei-ommended  hy  the  deimty  secre¬ 
tary  for  tnis  {uirposi-.  may  focus  on  cost  or  sclu'dule  risk.  I*  does  not 
require  an  explii  it  hu  us  on  technical  risk  or  jirovide  a  deiinitioii  of  li'cli- 
nical  risk.  (TicAt  K  is  dis<  ussed  in  di'tail  later  in  this  chapter. ) 

In  summary,  we  found  no  standard  dt'fmition  of  technical  risk  within 
nop.  The  otily  <lefiiiitions  that  do  exist  are  for  protti  am  i  tsk  as  a  u  hole. 
siH'cifyinn  cost.  sch<'<lule.  anrl  ix’rformance  as  three  comiKiuents  of  risk. 
Kven  tiu'se  delinilions  are  tiol  statidard.  however,  and  no  regulation  sets 
them  for  the  vvholi-  department.  (We  descriln.'  the  projtratn  offices’  \ar- 
ions  workinjj  didinilions  of  technical  risk  in  chapter  4. ) 


.•\pi>roaches  lor  assessinU  technical  risk  can  Ix'  either  quantitative  or 
qualitative.  de|H'ndinj:  on  w  hetlu-r  statist  teal  i>rohahilitii's  are  assigned 
to  a  risk  eUmient.  Bin  all  risk  assr*ssmcnt  entails  some  suh,te(  ii\  ity.  In 
virtually  all  approaches.  I'xpmls  are  iiski'd  for  suhi(s-ii\e  judjtmeuts  at 
what  the  risk  «‘lemenis  are  as  well  as  the  likeliluKvd  of  their  <k  i  urreiu  e. 
What  distinttuishes one  aiqnoach  from  another  is  the  inhi'nialion  that 
jt»K‘s  into  the  suhjective  jiidttments  ( such  as  test  results  or  iirofessional 
exjH-rtise)  and  the  ways  in  w  Inch  the  inl'ormation  is  obtained,  as  well  as 
the  kind  of  information  requested  (  for  examiiiv-.  a  .judgment  of  hitth. 
nu'dium.  or  low  risk  or  a.iudt^ment  alHiui  stalls. ic.il  prohahilitiesi. 


SiKS’ifical'.v  in  resjxinM'  to  lnitiali\e  1 1 .  tin-  l)efens<>  Systems  Manajie- 
tnent  ColleUe  puhhshe<i  Kisk  ,\ss('ssment  fechniques:  .\  llau'l'uook  for 
I’ro}tram  Management  I’ersonnel  '  iH'fense  S\stems.  IPS  ii  The  hand- 
IxKik  tJuides  protJram  management  offices  m  conduct intl  formal,  ((iianii- 
tative  risk  assi-ssments  w  it  li  \  arinus  |irohahih.>ti'-  approaches  It 
di'sc  'l>es  tfxds  and  te(  hnujues  intended  lor  deri\  i.id  huditcl  tiruri-s  lot- 
risk  that  can  he  use<l  more  s|x'cifu  iill\  to  quaniil>  technical  i  is!.>-  as 
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well.  Two  of  the  most  fatiucntly  iis«.-(l  qiiaiUitali\ v  a,>’  n.i  c  !,<  s  forto  li- 
nical  risk  assessment.  Ixitli  covered  in  I  tie  tiandbixjk  .<  e  i.-  "network" 
and  “risk  faetor"  approat  lies. 

Ttio  network  aiiproacti  involves  modelinfj  the  aeqiiisiiion  pnx  ess  for  a 
system  as  a  network,  in  wliii  h  the  nodes  or  end  jKiiir.s  represent  mile¬ 
stones  in  ttie  projtram  and  the  links  betwien  the  lunles  represent  activi¬ 
ties  that  must  1h>  carried  out  in  order  to  reach  each  end  ixiint  The 
probability  of  siicee.-vsfuliy  earr\  in>'  out  an  activity  is  usually  added  to 
the  m<Klel.  Numerous  computer  simulations  are  then  |K‘rformed  to  e\  al- 
uate  the  probability  of  achievinn  the  jtoal  represented  bv  the  network  as 
a  whole.  Kxamples  of  network  miaJels  are  the  "venture  evaluatioe.  and 
review  technique"  and  "risk  information  system  and  network  evalua¬ 
tion  teehni(|ue."  lv;th  of  whiclt  may  alvi  Ix'  list'd  to  address  schedule  risk 
and  cost  risk 

The  d^HXhS'.llltiyiproai  h  was  develojH'd  to  sup()ort  bttdjietinjf  for  tis  h- 
nical  risk.  In  ttiis  approach,  all  elements  of  a  system  and  their  assix  iated 
costs  are  identified  in  a  baseline  cost  estimate.  A  "risk  factor"  is  then 
determined  for  each  element  assoeiati'd  with  risk  in  the  weaixin  system. 
This  factor  is  a  number  by  which  tlie  estimate  should  be  increa.si'd  to 
account  lor  a  technical  problem  if  it  were  to  arise'.  The  estimate  and  risk 
factors  ar*'  determined  by  individuals  with  exis'rtise  in  the  technoiojty 
required  for  tin-  wea}x»n  system. 

Another  quantitative  at){>roach  is  det  ision  analy.sis.  Also  covered  in  I{i''k 
Assessment  Techniques,  it  n't|uires  the  di'velopnu'iit  of  a  decision  "tree" 
(a  kind  of  How  diagram)  in  which  sispiences  of  su[)[H»rtitij;  decision 
steps  are  laid  tuit  in  branchi's.  This  aids  in  i(lentifyin>i  uncenain  oi  <  ur- 
rences  in  the  chain  of  di-cisions  ITobahilist^IKThirinancejiuntilatuin. 
an  .•»i)proach  not  <-overed  in  the  haiulliook.  is  the  apjilication  of  a  com- 
putt'r  simtilatKin  to  v'qiiations  repn'setitin>;  factors  that  can  <  iintribute  to 
technical  risk.  Thes<'  factors  mav  lx'  •.,.<'<  ified  by  pivernmeiit  n'tiiiite- 
ments  or  derived  from  s|H'cific  system  ]H‘rformance  jioals. 

Such  risk  assessmt'iit  ai>t>roaches  as  these  can  Ih'  us<‘d  in  different 
as|K'cts  of  the  actttiisiiion  process.  The  jiro^ram  management  of  fices  can 
ii.se  them  for  buduetin>;.  as  in  the  use  of  Tiev  K  to  budifet  for  risk,  and  for 
day-to-day  projp  am  inanavtement.  as  vv  hen  tlis  isioiis  alxiut  program 
alternativt's  must  he  made.  The  ass<'ssments  can  also  tx'  used  in  de(  i- 
sions  math'  at  levi'ls  alxivc  the  provtrani  office,  for  l.oth  Iniditetum  and 
makinil  realistie  decisions  about  the  tei  hno!o)iy  of  the  \\ea|Hii'  system 
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Assessmonts  <»l  risk  can  also  lielj)  determine  if  program  milestones  have 
bt.'t'n  scheduled  appropriately. 


The  Defense  Systems  Management  College  handbook  focuses  on  quiuiti- 
tativo  approachc's,  but  qualitaftvo  techniqu-'s  are  perhaps  more  widely 
usc^d  and  are  generally  simpler  to  apply.  S/*me  qualitative  approaches 
provide  only  a  single  risk  rating  for  a  system  as  a  whole,  but  a  generic 
approach  recominenaed  by  LTN’  Aerospace  and  Defense  Company 
requires  a  comprehensive  examination  of  program  technical  risk  areas. 
It  involves  the  following  steps.  ( 1 )  IVvelop  a  decision  tree  to  display  the 
hierarchy  of  critical  system  requiiemcnts.  (2)  Specify  the  parameters 
for  tracking  technical  j)erformancc  during  the  program.  (3)  Review  the 
system  design  and  system  requirements,  pi-eferably  by  breaking  the 
work  down  into  its  essential  structure,  to  ensure  that  all  elements  arc 
examined.  (4)  Ustablish  written  criteria  to  define  levels  of  risk,  (o) 
Ensure  that  program  managers  are  aware  of  and  understand  the 
approach,  status,  and  n*sults  of  the  assc>ssment.  ((>)  fkicument  the  risk 
as,sessment  approach  and  results. 

Rather  than  using  the  probabilities  that  are  ('stimattsJ  for  quantitative 
ratings,  qualitative  approaclies  assess  risk  either  through  descriptive 
information  (identifying  the  nature  and  coinjHments  of  risk)  or  through 
an  ordinal  scale  ( high,  medium,  and  low.  for  example,  or  red,  yellow, 
and  green).  However,  qualitative  ratings  are  like  quantitative  ratings  in 
that  they  are  usually  based  on  the  judgment  of  cxix'rts. 


Another  apr.oacli  to  risk,  known  as  ri^jiiaiiagenient,  does  not  assess 
risk.  Risk  management,  tiecausc  it  idcntifi<’s  and  reacts  to  problems  as 
they  arise,  is  not  pros|xvtive  in  the  way  risk  assessment  is.  Risk  man¬ 
agement  is  th<*  implementation  of  strati'gies  to  control  or  moniu-r  pro¬ 
gram  risks,  and  it  may  follow  a  tcThnit  al  risk  assessment  and  fcKiis  on 
risks  the  itsM'ssment  idemified.  .Moreover,  risk  management  does  not 
necessarily  provide  explic  it  coverage  of  technical  risk;  it  may  center  on 
schedule  or  cost  considenitions. 

In  a  recent  effort  toward  risk  managcmieiit  in  a  jiarliciilar  phase  of  the 
acquisition  process,  non explu  itly  recogni/.ed  the  distinction  he'wc'eii 
risk  management  and  risk  assc-ssmemt .  non's  .lamiary  IP,  1P.S4.  dnvclivc' 
424h  7.  entiileci '  rransitioii  from  I)c'\elnpmeiit  to  Prodiic  t ior ."  rc'qiiirc's 
that  all  systems  in  de'.clopnicnt  •ind  jirodiu  tion  are  to  imiili  nK'nt  a 
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loi  iniil  pfo^rjiiii  of  risk  (‘Viilualum  :iiul  rrfliifiion.  I(  ciills  lor  tlio  jism'ss- 
mciii  of  jiroilriini  risk  tliroiijilioul  tlic  iiciiuisition  cyi  lc  iiiiii  cliiirUcs  pro- 
I'nim  iiKiiuiffomoui  u  iili  tho ovociilioii.  aiul  ilu*  with  the 

onforcriiu'iu ,  «if  the  provisions. 

The  rcsoiin  o  d<H  umrnt  for  imiiUMncntinU  tlu-  dirrctivo  is  i  ailed  ■'Solving 
the  ihsk  (-kpiation  in 'I'raiisitioninU  from  I‘evi‘loj>meni  to  I’rodiu  tion  " 
(lx)|i  tnamial  I'i-lo.T-M  i  and  was  develo|H><l  by  a  liefeiise  Si  lence  Ikiaril 
task  force  tinder  l.>e  leadership  of  the  deputy  eliief  of  na\  al  material  for 
reliability,  tnaintainahiliiy.  and  quality  a.ssnranee.  The  diK  iimi-nt 
iiK  lodes  a  s<  ries  of  templati's.  geared  to  the  most  critical  events  in  the 
desii>n.  test,  and  proiliution  elements  of  the  iiidiisirial  pro«-ess.  hm  it  is 
aimed  at  risk  inanaKenieni  and  d(K‘s  not  provide  a  technical  risk  assess¬ 
ment  ajiproach  for  pro>tram  manajii'inenl  offices. 

To  complement  '.Solvin;'  the  Risk  Ixinalion.  ’  the  task  force  de\elo|H‘d 
"Ik'st  Practices  for  Transitioning  Irotn  Development  to  I’riKiiu  tion." 
another  manna!  in  which  technical  .-isk  a.ssessment  is  recounized  as  a 
separate  function  i-ssential  to  tlie  .successful  development  of  a  weaixm 
,syst<-ni.  Tiir  ttutttiiu}  siifliivsis  ways  to  avoid  pitfalls  in  risk  manauemeni 
hut  d(K's  not  <lescrilK‘  or  reiommend  approachi^s  for  risk  assi'ssmen'. . 

In  iiddition  to  liKtkinU  for  s|K‘cific  approaclus,  we  bmked  for  more 
UeiuTii  definitions  of  and  eriiiTia  for  technu  al  risk  assi*ssment.  We 
found  that  Pop  has  not  estahhslu’d  a  j’eneric  definition  or  generic  cri¬ 
teria.  ,\ftcr  reviewinn  resi-arch  m  organizational  manaK<’nient  as  well  as 
risk  ass('ssm<*nts  by  ixipand  inivate  indiisirv  and  after  consiillini;  with 
a  niimtHT  of  exia-rts  in  technical  risk  a.ssessment.  we  develojxsl  five  i  ri- 
teria  for  di'fimni;  it:  ]>ros|>ective  assessment,  planni'd  priKedun-s. 
explicit  attention  to  t<‘<  hni<  al  risk,  diK  unientation.  and  re;cssessnn-nt  in 
eacli  actpiisition  plucsi-. 

If  an  assi'ssment  is  to  Im-  called  ■‘technical  risk"  asM-ssment,  all  five  of 
these  criteria  must  Ik*  prevni .  Kor  instance,  the  qualitative  and  quanti¬ 
tative  a[i[iroa<  h<‘s  we  des.-ribed  can  all  lie  used  to  ix-rform  teclmn  al  risk 
assessments,  but  iisinn  tln-m  does  not  ytnarantis*  that  an  asvssment 
im-ets  the  fiv  e  criteria  .X  ver.v  .soplhsi  icated  analysis  that  hail  not  l<e<'n 
diK-iiniented.  Idr  exanqile.  would  not  Ih’  a  tis  hnicai  risk  assessment 
under  our  definition.  This  is  because  an  undocumented  analysis  is  not 
very  useful  for  dei  isionmakinn.  ( Had,  of  these  criteria  is  discussed  in 
detail  in  i  hapter  d  i 
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l!»  suniinaiy.  many  IccImKal  risk  Jisscssmcnt  approac  lios.  quantitati\ c 
and  qualilativc.  art*  available  uitliin  non.  Hut  tlier(“  is  no  olTicial  ihiIk  y 
to  Kuide  pro>;ram  n»ana)J«TS  aiul  analysis  in  the  selec  tion  of  suitable 
approac  hes,  and  there  are  no  fjc-tieric  c  ritc-ria  defininK  an  adc-cpiaic'  tec  h 
nic  al  risk  assessnu'iit .  indei>c*ndent  of  eac  h  individual  approac  h. 


In  IflHl .  the  deputy  serreUary  c>f  iV-fense  eonducted  a  systematic  re\  iew 
of  iKin's  acquisition  prcK-ess.  with  the  objec  tives  of  rc'ducinn  costs, 
niakinj;  the  pren-ess  more  effic  ient.  increasing  proffram  stability,  and 
reduciiift  the-  time  rc*qinr<‘d  for  .s\  steni  deve  lopment.  From  this  revic'W 
evolved  '12  initiative's,  iiu  liidin^’.  for  c'xamjilc'.  the  use*  of  more*  c'c-onom- 
ical  prcKlii'  tion  rate's  and  earlier  tc'stintt  of  systems.  Initiative'  1 1 
requirc'd  the  sc'rvict'sfo  increase'  their  c'fforts  to  quantify  tc'chnical  risk. 
In  particular,  the  initiative  rc'qiiired  the  sc'rvict's  to  adopt  the  Army’s 
total  risk  a.ssc'ssinn  cost  c'stimate  'TRV'i;)  metlicKl  or  proiHise  an  alierna- 
li.e.  Kc'Fxirtin)*  on  the  status  of  the  initiative  in  a  .lime  8.  l!i''8.  memo¬ 
randum,  thedc'fiiify  .sc'cri'fary  of  Dc'fense  .statc'd  that  priK  edurt's  to 
budget  for  risk  had  Ikk'h  iinplc'mentc'd  by  tlu*  service's.  "This  initiative  is 
now  considerc'd  completc'd."  he  said.  Aftc'r  a  short  di'sc  riiition  of  trvi  k, 
what  each  service  ac  tually  did.  as  the  sc'rvicvs  rei>ont'd  it.  is  disc  iissc'd 
below. 

Tl  e  Army  dc'vc'loiM'd  the  total  risk  assessinjj  cost  c'stimate  methcHl  in 
1  !>7  l  in  ordc'r  to  be*  able'  to  add  an  incremental  dollar  fi>jurc'  to  t  he  base'- 
line  cost  estimate'  of  a  pronram  that  would  account  for  uncertain  evc'iits 
and  to  Ik*  able'  to  base'  a  justification  of  this  fiRiire  on  sound  estimation 
and  analysis.  The-  dollar  fiHiire  is  calculated  by  identifyin>i  iincc'rlain 
events  for  thc'  various  subsystems  or  comixinents  in  a  projtram  and  esti¬ 
mating  the'  amount  of  monc'y  that  would  tx'  requirc'd  to  cover  additional 
costs  asscxiattHl  with  eae  h  ixitc'ntial  problem.  Once'  these  costs  have 
bcTn  calciilalc'd  (by  nic'ans  of  various  tc'chniquc's  including  some' 
described  atxi’.e).  TRVK  orovidc's  an  c'slimatc'  that  rc'prc'sents  the  tr..de' 
off  bc'twcH'n  fundiii)*  only  for  ecisis  of  the'  program  that  can  Ih'  identified 
with  certainty  and  funding;  for  all  [Missiblc'  risks. 

Acc'ordint;  to  iRvt; ^iiidi'linc's.  the-  risks  that  may  Ix'  ine  hided  in  tkv  K 
calculations  arc'  dc'sinn  c  iianjti's  to  rc'solvc'  tc'chmcal  problc'iiis. 
re.-,c  hc'clulint;  to  rc'solve  tc'chnic  al  and  budjtetary  problc'ms  or  the'  1;  ' 

delivc'ry  of  c  cimixmc'iits  or  materials,  additional  tc'siinu  of  dc'si>;n  ecu  rc-c  - 
tioiis  and  hardware'  to  siiiqxirl  them.  nonnc'Hht.'c'nt  human  error,  and 
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projliain  termination.'  Many  of  iliese  risks,  of  course,  arc  not  neci  ssa!  ily 
tci'linical  in  orifiin.  Tims,  to  fnlfill  Initiative  1 1 .  analysts  nsmii  tlie  iiiv  K 
priK'cdnrc  (or  any  alternative)  must  (listin)(uish  ti'<  Imical  risks  from 
other  risks  and  th<-n  quantify  the  technical  risks.  One  vva.\  to  do  so  is  to 
estimate  nunuTical  probabilities  for  the  (Hcurri>iu-e  of  vai  ions  tet  hnii  al 
problems.  In  network  analysis,  the  probabilities  are  ustsl  as  injuit  for 
cak  Illations  of  overall  technical  nsk.'l  hey  can  also  serve  as  a  basis  for 
projecting  the  cost  implications  of  each  problem.  A  sc'cond  and  mon- 
direct  way  to  quantify  tec  bnical  risks  is  sinqily  to  e  stimate  the  amount 
nt'eded  to  cover  ea<  h  |>ossible  probU-m  and  use*  this  amount  ;ls  a  quanti¬ 
tative  indic  ator  of  risk. 


\rmy 


S'avy 


Orijtinally.  TH.V  K  funds  were  calculatc'd  for  the  jirc'jinKluc  tmn  phavs  of 
syslem  ac  quisition — rc'sc-arch,  develoi>ment,  testiii);.  and  evaluation  — 
iK'cause  nine  h  of  thi*  risk  assoc  iat{*d  with  weaixm  systc-m  develojiment 
arises  in  tlie  <*arly  stajjc's.  In  its  intc'i  nal  budnetinn.  the  Army  now 
api'lies  rit-V  K  to  the  lUCKluction  ))ha.se  for  some  systems  as  \cell.  The* 
•Arm.v's  r<*s(Kin.se  to  Initiative  1 1  was  to  continue  the*  i>ri'vioiisly  insti¬ 
tuted  TK-VK  program.  I’rojtram  offievs  were  not  directed  to  distin>;.iish 
tc'chnical  risk  in  th<*ir  Ttev  K  anal.vses  or  to  quantify  the  costs  a.sscK  laied 
s|K*cifically  with  technic  al  problc*ms. 


liesicoiidinjt  to  Initiatice  1 1 .  the  Nav.v  eslablishecj  a  pilot  pro^r.im  to 
evaluate  the  use  of  Tiev  t.  u ii h  si\  systc-ms  I’he  oiMiuon  of  the*  cikh  ill- 
nator  within  the  Naval  .-Xir  Sysic-ms  Command,  where  the  pilot  program 
was  sc*t  uji,  is  that  the  method.-,  for  calculatirjt  risk  lumK  aie  so  compli¬ 
cated  and  require*  so  mucli  time*  t  hat.  when  the*y  are  alforil.ihle,  ihey 
must  Ik*  done  by  outside*  exjK*rts.  c’onsee)ue*nt  ly.  he*  staled,  i  lu*  oulsid'*i> 
lK*come*  the  risk  ex|K*rls.  and  (iroy.ram  manai;e*rs  ftain  hit  li-  know  ledyi- 
The*  Na.vy  has  confinc'd  Titv  K  to  pre-production  (ihases  and  has  ni'M  i 
move*el  iK'N'oiid  the  pilot  effort,  .'■smie*  of  the*  systc-ms  in  the  pilot  jiroeram 
ha\e*  drop)K-d  t lie*  use*  of  TltvK  and  others  are  no  lonifer  elif.;ili!e.  ha\  im( 
move*el  inio  priKiuciion.  The*  (>ilot  effort  die!  not  recpiirc  i  hai  nev  K  aii.il- 
\  ses  |ia\  e\phe  It  alti-ution  to  tc*chnical  risk 
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DOD'i  PoUci^  for  TcK^hnic*!  Rl»k  AMTMimrnt 


Air  Force 

The  Air  Force  chose*  not  to  ado|>t  TRvi  i:  for  dealing;  with  risks,  and  there¬ 
fore  none  of  its  pro>'rain.s  has  trvk  fiindiii;f  The  ri’sjxuise  ol  the  .\ir 
Force  to  Initiative  1 1  was  to  state  its  satistactioii  with  the  cost  estima¬ 
tion  proc»*<lures  already  in  us<*  for  quantifying  risks,  sav  ing;  that  it  saw- 
no  advantage  to  the  mvcK  apjmoach.  The  Air  Force  issued  no  reiiiiire- 
ment  for  explicit  attention  to  technical  r’sk  in  those  iiiik  edures.  Initia¬ 
tive  1 1  thus  changed  no  Air  Force  fxilivies. 

Initiative  1 1  and  the 

Defense  Systems 

Acquisition  Review  Couttcil 

According  to  the  director  of  major  systems  acquisition  in  the  offn  e  of 
the  undersecri'tary  of  Ik'fensi*  for  res<'arch  and  engiiieiTing.  Initiative 

1 1  led  to  no  cliaiiges  in  priK-eduri*s  or  diM-iimentation  that  the  iis.vKc  uses 
to  ev  aluate  t  he  d«*velopment  of  systems 

Summary  of  Initiative  1 1  initiative  1 1  was  mtendeU  to  promote  tlie  quantification  of.  and 

biidnetinjt  f<»r,  te*  hnieal  risks.  In  resiKHisc  to  Initiative  1 1 .  one  Navy 
eommand  (the  Naval  Air  Systems  Command  i  tried  a  small  i itv  k  jnlot 
program.  Tlie  Air  Force  made  noelian>;es  from  the  oiits<>t.  and  llie  .\rmy 
has  main'aineil  the  mVK  pro):ram  at  its  earlier  status.  Vet.  as  we  noted 
earlier,  to  fulfill  lnitiati\e  1 1 .  tlu'  s«‘rvic<*s  \M>uld  need  to  i  ondiict  anal¬ 
yses  that  distintttiish  technical  risks  from  other  risks  and  qii.mtit'\  the 
technical  risla'  'w  means  of  prohalnlity  ormst  estimates.  iidix's  not 
nec<'ss;irtl>'  do  so.  and  none  of  tlie  .s<>r  v  ices  has  instructed  its  program 
officisi  to  use  Ticv  K.  or  ativ  alternativ m.  in  wav  s  i)ial  would  deal  sjM-cili- 
cti'ly  with  te<  hnical  risks  Nor  lias  the  nsvac  adopted  anv  iirocedure  or 
revpiiremi'nt  that  would  entail  dist int;uishin>i  and  qiiantifv  in^;  thcM* 
risks  The  net  effect  of  Initiative  I  1  on  technical  risk  assi-ssinent  )>roce- 
dures  has  thus  Ixs  n  neniinihle. 


Siinuiiary 


We  fouiul  that  the  l>eparlmeni  of  iV-fenv  has  ^teiieral  [m)!icii-s  callmit  for 
te<  hnical  risk  assi-ssment .  hut  the  p-.ilicies  do  not  prov  ide  any  standard 
I  •op  definit  loll  of  iiioitram  risk  or  t  is  hnical  risk,  and  t  hev  otter  no  rinid- 
aiice  for  desutmiiu  or  si-lectintJ  suilahle  asM>ssment  approai  hes  Iti  itula- 
tioiis  ^;overiiin;t  sxstt  m  documentation  require  that  techim  al  risk  he 
addressed  hut  do  not  deline  lei  him  al  or  (a  oiti  .im  risk 

Nclt her  the  tisvlic  nor  tlie  serv  ices  hav  e  res|Mrnded  to  lull  la.  .v  e  I  I  hv 
requiring;  assessments  that  disiutituish  tei  him  .d  risks  lioin  oilier  pro¬ 
gram  risks  or  qii. lilt ifv  the  technical  risks 
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•  Differences  in  How  the  Program  Offices 
.\ddress  Technical  Risk 


In  this  chaptor,  wo  di-scrilH'  how  the  2o  projtram  nianajtcmeiit  offat's  wo 
oxamiiu'd  attempted  to  identify  tfu*  tet  Imical  risks  of  tfu*  2'>  systems. 
IkTauso  of  Initiative  1 1  and  tlie  Ik'fense  Systems  Manarti  nieiU  Colle)te 
handlxKik  on  risk  assessment,  we  exiHstr-d  to  find  tlie  offn  os  assessinf? 
Uahnioal  risk  in  quantitative  (or  probabilistio)  tenns  and  earmarking 
funds  to  cover  tliat '  '.sk.  Iks’auso  of  ixiii  nxiiiiromonls  for  milestone 
reviews,  we  als<i  exiHsKsl  to  find  diM  iiments  explainin}>  how  risk  is 
assessed  and  fiow  the  amount  of  ftinds  nts'ded  to  cover  risk  is  calcu¬ 
lated.  We  iK'lievisl  that  some  iiffices  mi^lht  iiientify  risks  in  other  ways 
as  well,  perha[>s  usinjl  qualitative  ajiproac  lies  like  thos<’  described  in 
chapter  2  or  scttifif*  up  a  risk  management  system  to  pinjxiint  technical 
problems  as  they  arise. 

In  short,  we  exfK'cted  consiilerable  variability  in  approai  hes  to  tcshnical 
risk  and  wanted  to  lx*  sure  that  our  data  collec  turn  did  not  miss  this 
variability.  Hence,  in  our  int<‘rviews  and  dixument  reviews,  we  investi¬ 
gated  every'  effort  of  the  program  offices  to  idimtify  technical  risks.  V.'e 
have  us<'d  the  expn'ssion  “risk  effon"  to  refer  to  whatever  aiqiroach  we 
found  in  the  2')  offices,  |•es<'r^•ing  the  term  "tc'chnical  risk  asM.'ssmcnt" 
for  efforts  that  met  the  particular  criteria  dt'scribtsi  bc'low. 

In  this  chapter,  we  cover  evaluation  questions  4-0: 

4.  What  are  tf  e  characteristics  of  current  efforts  to  identify  the  tech¬ 
nical  risks  of  new  systems? 

5.  How  are  efforts  to  identify  ti“chnical  risks  implemented? 

0.  What  information  on  tcThna  al  risk  is  available  to  decisionmakers  in 
the  review  pnxi'ss? 

To  answer  question  4,  we  first  discuss  the  niimlKT  of  program  offici-s 
that  us«*d  quantitative  efforts  to  budget  for  risk.  Then,  to  prov  ide  a 
basis  for  desenbing  efforts  in  all  25  program  offices,  we  establish  five 
criteria  that  are  ess<*ntial  in  tin  l;ni<  al  risk  ass«>ssment  and  discuss  the 
niimlx’r  of  program  offic-s  meeting  Ihi'se  criteria.  To  answer  (juestions  5 
and  0.  we  lonsider  all  efforts  we  found,  whether  or  not  they  met  all  five 
enteria. 

.'\n.swers  to  a  few  study  qui-stions  from  resi»ondents  inside  an  offii  e 
were  inconsistent  in  ways  we  could  not  ri's<ilve  by  referring  to  the 
majority  answer  or  jirogran  do<  umeiils.  Other  information  we  needed 
was  simply  not  available,  and  \\  here  tins  is  relevant,  «c  note  it.  Kor 


34 


(f  AO  I’HMI)  TrrhnKal  Rit»k 


i'tiApfrr  3 

DifTrrt’nct^  in  l{<m  ihr  Pn>f(rfttn  Offlmt 
AddrrfM  Tf<Knir«l  Kii^k 


nuist  (tf  (Uir  qm'stions.  Ihouf^h.  unovftall  ri‘s|X)ii.s<‘ cinild  iMM  ixird  fni  ;itl 
(ir  ;ilnii>st  all  the  iiHici's. 


/hat  Are  the 
liaracteristics  of 
urrenl  Efforts  to 
ientify  the  Teclinical 
isks  of  New  Systems? 


'I'd  answer  this  es  aluation  qiHSitidti.  we  (irsl  {IcscrilK*  ()iiari(ilali%<‘ 
cUdtls  Id  huduct  Idf  risk  aiui  then  tlt'sniln-  llu*  c-lldils  wi-  hniiul  in  all 
dtfiei-s. 


Uantifyinj?  atld  liudj^cling  IK-Spitc  the  availahilny  of  tlu-  Defense  Systems  Manaytement  Cdlle^ie  risk 
•r  Technical  Risk  Jissessment  hatulhiKik.  and  <lespite  the  <le;MiIy  .s<tTetar\  's  asserfidu  Ihal 

Initiative  1 1  has  Ks-n  imiilementJ^!.  noiv'  of  tlu'  offiii's  we  exatiiined  had 
{KTfonned  a  quantitative  effort  and  used  it  for  the  pnrjiosi'  siH'eiiieii  in 
Initiative  1 1  — to  ealeiilate  the  fiinilinn  neei'ssary  to  (over  leehnieal  risk. 
One  office.  res|)dnsihle  for  the  Army's  ShnrlXanjte  Air  Defense  (’om- 
mand  and  Control  system  ( si  iota  t  »(.>).  did  iierform  a  quantitative  assess¬ 
ment  of  technical  risk  hut  tlien  siipjxirli'd  its  application  for  risk  funds 
with  an  entirely  different  ass<*ssment.  The  latter  asM'ssmenl  used  ruv  i; 
to  calculate  cost  risk  from  jioienlial  sclu'dnli'  slippa>:es.  in  w  hich  lei  h- 
nical  risks  wi  re  not  quantified  or  even  explii  itly  considered. 


SSCSSmcnt  Criteria  and  Alihou>;h  we  found  that  no  quantitative  efforts  had  heen  us<m|  idr  risk 

isk  Efforts  hudnelinn.  we  found  other  efforts  in  all  Ll")  projtram  offices  and  (olhvted 

descriiitive  information  on  tliem.  We  imixiM-d  no  definition  of  "risk 
effort"  hut  simply  askinf  res|H>ndent.s  lodeserilH'  relevant  aetivun-s 
however  they  defined  this  expression  'f  any  part  of  their  (dfort  had 
Uh'H  handled  hy  sources  outside  i!.e  o'lfici — for  example.  ser\  i<e  lalxi- 
ratory  staff  or  contractors — we  interviewml  these  sources  as  well. 

As  we  reixirti'd  in  chapter  2.  ixm  has  no  [xiln  >•  calling;  for  a  particular 
assessnienl  afuiroach  or  siH'cifyinft.  in  >;eneral  terms,  wh.it  sorts  of 
a.ssessmenl  are  acceptahle.  Since  we  'ould  not  compare  the  elfoits  we 
found  to  any  official  lx»i>  standard,  we  re\  iewixl  the  researcli  on  oiytani- 
zalional  management  as  well  as  risk  iLssessments  <  cinducti'd  in  l»ili  and 
the  defense  industry  (itiven  in  the  hihlio^raphy  )  and  consulted  meiluKl- 
oloflisis  familiar  w  ith  the  area  From  this  rev  iew  ,  we  develoix-d  live  i n 
leria  Ihal  can  lx-  considered  l■ssenllal  in  the  assessment  of  lechnii  .il  i  isk: 
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in  -'low  thr  Offlcr* 

A4drrM  TrchiticAi  Rl»li 


1.  prosjM'ctivo  ;LSM‘ssnii‘nt:  I’ossihlt'  fiitiiri*  Itvhnical  prublanis  arc  ion- 
sidoro*!,  not  lurrcni  problems. 

2  plannndj)i^(Hj^i!ia“s;  AsM'ssinfiU  is  planm'd  and  syslfinatic,  not 
inndi'tital. 

3.  atf('ntii>n_t()_tt‘tlini(  al  n  TIuto  is  explicit  a'trntion  to  t»*<  hnical  risk, 
not  just  to  schedule  <»r  <'ost  risk  Witli  cons ■d(‘rat ion  c)f  technic:.!  risk  left 
implicit. 

4.  documentation:  At  a  minimum,  t«x  hnical  r'sk  assev.meni  pnK  cdure.s 
and  results  ;ire  written  down  in  vime  form. 

.o.  nsisM-ssnu'nl  in  each  a;  qiiisition  phase;  .\(‘W  or  u[Klated  sments 
are  n.ade  in  order  to  d.  tet  t  changes  in  risk  durmR  a  system's 
devel<»pment. 

Those'  ( riU'ria  are  not  nect-ssarily  definitive,  hut  they  do  reflect  relevant, 
attainable  characteristics  and  thus  provide  a  reasonable  basis  for 
appriiisiiiR  the  cpiaii'y  of  risk  efforts.  .Moreover,  siiue  wt'  did  ni>t 
attempt  to  puiRe  the  lucuriuy  of  risk  ratitiRs  or  the  suitability  of  jiartic- 
tilar  asvssment  approac  hes,  thes*'  five  c  riteria  repri'sent  a  minimum 
standard  of  qualit.v.  ,\.s  wc*  noted  earlier,  wi*  ri'sc'rve  the  term  ’  tec  hnu  a! 
risk  asM'ssment  "  for  efforts  ntc-etiiiR  all  five  c  rileria. 

Jtelon .  we  briefly  diseuss  eac  h  of  thi'  five  criteria  and  thi-n  c  iie  the* 
numlwr  of  program  oflices  with  risk  efforts  tlial  met  eacte  •im-  Then  ue 
discuss  c'fforts  mec'tuiR  .ill  fivi-.  Ti'!;ii‘  3.l  shows  the  criferi;;  inat  were 
nvt  for  the  2')  systems  in  table  I  I. 
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•  Technical  Risk  Aaaattmant  Criteria 

Servicr/  and  tytlem 

• 

> 

1 

• 

o 

a 

■■PDMHH 

Documented 

Planned 

Rea 

Explicit 

■tested 

In  each 
phase 

Army 

AHIP 

X 

X 

X 

X 

M1A1 

X 

X 

X 

WLRS/TGW 

X 

X 

X 

RPV 

X 

X 

X 

X 

X 

SHQRAO  C2 

X 

X 

X 

X 

X 

Navy 

ALWT 

X 

X 

X 

ASPJ 

X 

X 

X 

ASW  sew 

X 

X 

X 

X 

X 

ATRS 

X 

X 

CV  HCLO 

X 

X 

X 

HfAJ 

X 

X 

JTIDS 

X 

SUB ACS 

X 

X 

X 

T45TS 

X 

X 

X 

X 

TfKJcnt  II  (05) 

X 

X 

X 

V  22  Cip'f  y 

Air  Force 

- 

X 

X 

AVMAAM 

X 

ASAT 

X 

X 

X 

X 

C  17A 

X 

X 

X 

1  S/A  AVPf. 

X 

X 

JTIOS 

X 

X 

JSTARS 

X 

X 

X 

Walk  XV  ItF 

X 

X 

X 

X 

NAVSTAR  User  E()ji|prnenl 

X 

X 

X 

SRAM  II 

X 

X 

X 

Total 

16 

9 

18 

18 

15 

f>s|ii‘(1ivf  As'wssn  •-•lU  T<»  Ik*  rvfiil  jirtHlu  tivi'ly.  tiH-hni*  ;!!  risk  a>M‘ssnii’iit  innst  iilcnlify  risks 

well  tlicy  ;u  iiial  |.r<il>!«-iiis  An  ass<-ssnti‘nt  carU  in  (lie 

(It’vrldpnuMit  imK'i'ss — listing;  risk  arcits  and  iH-rliaps  csi imal in>;  dcjirifs 
of  risk  as  ««'ll — »an  provide  a  sysicmatn  Inundation  lor  fiirtlu-r  anal¬ 
ysis  and  revision  as  a  svsi«-ni  mov«*s  iliron^li  acmnsition.  Hut  an  assc-ss- 
inen»  based,  for  evainple.  on  ^^•sts  eomliieled  jiisl  prior  In  i  he  priMlini  ion 
derision  (  milestoiU'  III ;  <Uh's  not  ass<‘ss  the  risk  that  the  prohlenis  w  ill 
(H'l'iir.  It  uneoM'i  s  the  fai  t  that  prohletns  have  alri’adv  iHnirred 


Pmgr  .17 


(•AO  PKMI>  MV  1%  TiK'hnU'Al  Ri^h 


I’llKI-dlll'CS 


>li«  it  Attciilion  III 


i  hiiptrr 

IhfTrrrur^Hi  in  ifw  rn*i(r«fti 
Ad<lrT»M»  IrchniriU  Ki«li 


l*rii>[n'ri i\ r  n^k  wi'ii-  Kiixliii  ii-cl  lur  111  (m  I'p  1  ihh  i-iil  i  hI  IIh' 

ss  sii'ins  h'nr  t  111'  !*  Ill  liiT''  I  III'  tii  (KTi  I'cil  J.  (I’l  hm<  ;il  |'M»t'li'ii;s  \\<  i  r  idi'ii- 
tifii'il  Its  ihfV  iiMisc.  (iiirn  tliiiiu^h  risk  riiaiiiiki'iiii'ii!  sirms.  but  i  isks 
wcrt-  ii(i‘  Kifiililiftl  lit  ;ul\uii<f 


ri'i'!'‘i.ii'al  risk  assc-smi-iils  ntU'*i  1h' » ari-lulK'  plaiiiii'il  -  that  is.  risks 
must  Ih.'  Klfiitificil  by  tlflilH-raU'.  systfiu.il  u  priK  I'lliirrs  W’li  Imut  plait- 
iiiii^;,  Ufliiiual  slat!  ruciiilH'rs  may  ovfiliHiR  imUMilial  ri-ks.  ur  sniiii-  may 
iM'lii'Vf  a  svstcm  s  rumiMiix-iits  in  tw  hi>;b  m  ri'k  w  bil''  i.i Iuts  ImIh-m-  i  br 
samt*  ctimiKiiii'uis  in  Iw  miKirrati-  or  low  m  risk  Sm  b  ilisi  n-pam  u-s 
fotiM  I'a.sily  >t<i  iiiircto^tm/cd  until  a  risk  tiirmsl  iiiio  a  ittajor  priiblrm 
'I'i't  limcal  risk  itsst'ssmi'm  t  atinot  fonsist  of  only  iinpl.inni'il.  (h  <  asional 
•iiviisMons  of  risk  in  staff  mts-tinjts  or  oilier  ail  Ihk  |>roi  i  ilun-s 

We  foiiiui  IH  systems(72  |K'reenI )  with  planned  elfoiis  ,\(l  Imh  efbuis 
wero  made  for  7  (L’H  ikti  enl  r.  risk  uas  eonsidered  \v  ben  siaM  iiiemln  rs 
oi' outside  entitles  liroii^lit  it  up.  hut  risk  ef foils  were  not  a  pl.inned 
aeti\  iiy. 


Tis  br.u  al  Ihsk  Siiue  assessments « ombme  the  tishnie.d,  eost .  and  si  bis  lull  i  oiui>onenls 
of  overall  program  risk  For  <'\ampie.  the  .\rmy  s  i  av  i.  |iii h  edure  us«-s 
■'lii;;h."  "low,  "  and  inirsi  likely”  t  ost  esiiin.ites  for  eat  b  suns\sieiu.  pro- 
diiein^;  an  o\erall  estimate  of  <  osi  risk  lor  t  be  s\  stem  as  a  u  bole  1  be 
vuirees  of  subsystem  eosi  risk,  ini  luiiim;  |sissible  tei  bull  al  problems 
may  not  Ih'  uienlifiisi  explu  itl\  .  il  not.  Ibe  assi-ssineoi  ill  noi  !«■  uselul 
as  an  indieator  of  tlie  system’s  tis  hnit  al  risk 

In  our  study,  risk  elforts  lor  IS  systems  (72  (len  ent  i  itleiiiilied  tei  bnu  al 
risks  exjilieitly  I'dlmts  for  the  7  otliers  (  2.S  |HTeent  1 1  onsulered  iis  b- 
nil  al  risks  only  inipln  ill,\ .  in  first  risk  or  m  bedule  risk  a  isessinents  or 
measured  overall  pro^tram  risk  w  ithout  is4ilatinit  its  i  om|miienl  o!  its  h- 
meal  risk. 


Teehmi  al  risk  assessments  must  Ih-  iIih  umentetl.  so  that  (iio;;i.im  man.e 
iters.  t(s  linit  al  sialf.  and  re\  lewers  t  an  monitor  the  jiroi  etlun-s  lollo  osl 
to  identily  r.sks  and  t  an  \er>l\  the  results  I  bis  i  .ipabilily  is  es|).-i  iall\ 
im)>or1ani  for  (iroi;ram  maitaiters  anil  sial  I  new  l\  assn^nisl  to  an  on^om^; 
di'\  elopMtent  el  liiit  and  for  mili-stone  re\  lewers  w  bo  mikht  lus-d  to 
know  s|Ns  die  details 
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(iiK  iinuniiil  I  I  IT  I  Ilf  m.i  |i 't  it  \  i*  1  ih-m  f  n'  i  !fi  liim  ;il  i  w  is  ;iiiilri->-,ii| 
111  si  .il  I  iiifi  !  iii^;s  .lint  |ii  I  im  .III'  i.l.iiiiiiin',  ii  III  ml  rt'i  nnlim;  1 1  n-  |  n  i  h  i-.s  i  ii 

1 1 -s' ill  s  I  III  1  lifsf  1  .tsfs,  iiiir  it.ii.i  up  risk  f  I in  s  ucii-  ulit  mmii  1 1  uiii 
irif  I  S  ifis  s.  .IS  wf  Ml  ill'll  in  1  li.i|iifr  1 .  m  il  li  pini^r.iiii  nl  I  n  r  si  il  I  i  All  I  In’ 
|iri>)4i;iiii  111  111  i-s  |iri>\  nliil  sinnf  risk  inliiiin.ii  inn  in  I  lif  niilfslnnf  ii'\  ifW 
ill  in  I II 11  111  s,  lull  It  u  .IS  iiisiM  I  ii  If  III  l  Inr  l  f  ;is<  ms  \\f  ‘Ifliiifiilf  in  1 1  if  sfi 
tiiiii  Ik'Iiiw  nil  If!  linn  .il  risk  intnnnai mn  ;iv  iiii.ililf  In  dn  isiniiniiikfi  si 


LSSI~VSIIH‘II|  in  h^li  II  .\n|llls|(l<in  MiiVflllfllt  Ilnlll  nllf  ;il;,is<|  to  liif  IlfM  Is  li.isfd  OM  I  Ilf  sl.il  ns  i  iI  ;i 
iLs«-  s\s|fni  s  (fi  tiiin.il  [irntilf  ;iis  l  ln-..  |irnj;r;fn  in.iii.iu.  iiif ill  sl;ill  ;ind 

rf\  If  wcrs  iniisl  U-  .ililf  In  i  r.n  k  ili.-  idf  nl  il  u  ;il  mn  of  risks  dim nj;  .i 
s\  s'f  III  s  (If  \  flniMiifiil  and  tt.iiinf.  irotn  d.ii.i  sm  li  as  ii-si  ifsiill  s  nr 
f\|ifrl  iiid'^iitf III .  liin%  linn  II  i>rii.;rfss  li.is  lns’ii  inadf  llisks  iiiiisi  lirsl  In- 
,isvssi'<l  »'arl>  III  (iiiK  f|il  f\iilnr.ilinii  .nid  lln  ii  I'f  ri-assfssi  d  l.ilnr  111  (Ilf 
sann- ptiaM'.  so  ilial  d«s  isinniiiakfrs  at  iiiilfstoiif  I  t  an  kiuns  itlial  risks 
lia\f  Ixfii  nifi.ldiftt  and  lif\\  iinn  h  prii>ir<"ss  h.is  ln-fn  iii.idr  inMaid 
llifir  rfviliitinii  Siin  f  systfin  dfVflnpiiifiii  Is  nnCiiin>;  .mil  inilfsiniif 
t'fS  if^Ns  in.iy  If.nl  Pidfsiijii  i  li.nnlfs,  .nintlifr  ass«-ssinf ill  wn'ild  lif  diif 
•  •ally  III  llif  iif\t  pli.isf  Tlif  saiiif  ln|;n  snpjMiris  Innti'T  rr.issi-ssinfiit. 
If.nlinU  111! nil, Ilf  1>  in  ihf  nnlfstoiif  III  iln  ismn  Inr  pind  n  i  mn  .uul 

(If  (lint  (Ilf  Ilf  III  .III  o(ifratjnri.d  s\  stem  I;i  shnn.  ti  <  tiim  .ii  i  />•)•.  .isscss- 
n mills  slmiild  (if  I  nfidin  I'll  al  least  l  n  n  e  in  l  ai  |i  .n  i(ii|sil  mn  pli.ise ,  niie 
carK  and  .iiintlmr  1  ilc ,  to  snppoii  si.il  I  .md  ''A  m a  dis  ismns  riaJ.irdinv; 

\s  lift  Ilf  r  and  Imvs  In  |i!im  <md  •  Mif  iif  \I  pli.ise  K.n  li  i  f.is.j'ssinrnl  nia  v 
I  S'  an  fill  Iff  l>  Ilf  A  f  I  ini  t  ni  ii|si.i!f  I  lif  p!  f  \  Inns  mm  I-  I'Jii  I  f  1  1  df  pn  1  s 
I  Ins  <  I  lift  mil 
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Figure  3.1:  The  Criterjon  of  Reb^sessmeni  in  Each  Acquisition  Phase 
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\\V  wri  c  im;il>lc  i<>  <1(1  ;ii('  a  t  itnc  lor  carl;,  and  lal(  ass<'sMii(  nt  in  cai  h 
ai  (lui'.ii  Kin  |iliasc  I  nr  (mi  h  s\  si  cm  I’r  niii  am  management  dlTiees  skip]  i'  •(! 
al  least  line  milesiKiie  |(ir  man\  systems  I'of  nI  iiel  s.  Iks  allse  (Ie\ elep 
inent  had  heami  se\era!  mmi  s  acu  m-  imleslnne  tex  tew  dates  had 
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slipp<*(l,  \vc  could  not  cstablisli  carl>’  or  laic  limes  with  i^rccision.  An 
alternative  approach  was  simply  to  ask  wliether  risks  were  assessed  at 
least  twice  in  ea<  h  phase  by  means  of  annual  uiKl.ites  or  the  like.  We 
could  nut  be  sure  that  there  had  been  one  early  and  one  late  risk  effort 
in  each  phase*,  but  we  could  determine  that  one  occurred  later  than 
another  and  that  the  program  management  staff  therefore  had  had  an 
opportunity  to  dett*ct  any  changes  in  risk. 

Since  many  of  the  programs  in  our  study  skip|x>d  one  or  two  early 
phases  or  had  not  yet  reached  full-s<  ale  developmer.',  we  determined 
whether  a  program  management  office  had  assessed  risk  at  least  twice 
in  each  phase  a  system  had  reached  and  not  skipix'd.  Of  our  2')  offices, 
1  .'j  ((10  percent )  had  done  so.  Most  of  t  hese  offices  ( 1 2  of  the  1  o )  fx'r- 
formed  risk  efforts  as  an  ongoing  part  of  program  management. 


Risk  Ivfforls  .Meeting  Ail 
Five  Criteria 


Only  3(12  peic(“nt)of  the  risk  efforts  pt>rformi*d  fur  these  .systems  f;.l- 
filled  all  fiv*‘  criteria  (as  we  showed  in  table  3.1  );  the  .Army’s  Remotely 
Riiot<“d  Vehicle  (Ki'V)  and  Short-Range  Air  Ik'fense  Command  and  Con¬ 
trol  .System  (siioittP  cj).  and  the  Navy's  Antisubmarine  Warfare 
Standoff  Weaixm  (.vsw  s<w).  The  prosjx'ctive  decision  risk  analysis  for 
the  lil’v  was  conductixi  ai  t  ording  to  a  planned  schedule,  first  in  1081 
and  subsequently  in  thr<-e  annual  ujidates.  The  1081  and  1082  analyses 
fiK'Used  exiilicitly  on  tec  hnical  risk  For  the*  1082  analysis,  staff  mem- 
IXTs  were  asked  to  rate  each  of  the  Kl'v’s  subsystems  (target  Icxation, 
air- vehicle  endurance,  and  .so  on)  on  a  six-jMiiiu  scale  of  technical  risk, 
ranging  from  "none  or  very  low"  to  "unacceptably  high."  The  ratings 
were  anchored  to  quantitative  estimate's  of  failure  and  verbal  dc‘si  rii> 
tions  of  risk.  Systems  analysis,  assigned  to  the  program  management 
office  according  to  a  matrix  organization,  aggn-gated  ratings  from  indi¬ 
vidual  staff  im-mtiers  to  arri\c-  at  overall  qualitative  ratings.  Dexumen- 
tation  dc-scribcd  the  ])nwess  and  results  in  detail.  (Sc-c  table  3.2 ). 
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Table  3.2:  Technical  Risk  Rating  Scale 
for  Remotely  Piloted  Vehicle 


Qualitative  label 

Probability 
ot  failure 

Description 

Nore  01  .01/  'oa 

0-  4L 

r u"/  dc.ciopod  and  in  product  t,n  moots  l■■■.lltdry 
spocilications 

1  OA- 

5-16 

Fully  developed  arid  producible  but  not  yet  m 
production  meets  military  speciticalrons 

X^OdOMtO 

16-30 

Needs  lilllo  (uriner  dovolopnieni  his  not  met  military 
spccil  cations 

High 

31-40 

Needs  (uriher  development  and  debugging 

Vci  /  h.  gh 

41-50 

Has  lx;en  designed  but  needs  entcnsive  development 

Lhi  iccplahly 

hiuh 

51  -1 

Is  theoretical  and  may  enceed  the  state  ot  ttie  art 

Source"  w  B'.XJcJC.'n  ft  a!  0<;f.*v*on  Ar.ai,^,is,  P'lQlcO  V«;ti»Ck  tSt  LOu  t»  Avial'OO 

Rcseatcn  af*o  Dt'vt*ioo^‘fnt  CorrifT'jno  19^2)  p  B  J 

Tl.c  pritnr  <onli;u  tor  lor  the  asw  sw  provided  pn>s(H't  tivc  risk  assoss- 
monls.  locusinn  explicitly  on  le<  hiiical  risk.  .Indytmenls  of  the  likelitKKul 
of  th<*  systi'in's  ineetinn  ix'i  formaiu  e  retimreinenls  were  collected  from 
iniine  eontrticlor  and  snheonlrai  toi'  staff  and  then  diK  uinented  in  the 
form  of  qualitative  ratings  of  risk  (hinh.  moderate,  and  low).  Ueassi'ss- 
ments  were  conducted  renularl\  and  according  m  plans  nenotiati'd  wii  (i 
the  projiram  office  staff. 

The  siioic\l>  CL'  projiram  oftice  htuulleti  its  risk  assessnumts  in  different 
ways  as  th(' systein  mo\ed  ihrocifh  di‘veloi>ment.  Forthe  IHSo  reasvss- 
ment,  the  t)fo>tiam  oftice  hrou^ht  in  a  sui'inn  t  contractor  to  collect  (eo- 
siH'clive  risk  data  from  pro>;ram  en)fineers  and  other  sjx'cialists.  The 
dat;i  f<K  Us«-d  explicitly  on  tei  hnii  al  risk  and  wme  <‘Xpres.s<'d  as  (piantita- 
ti\  e  ratings  of  tm-  probability  of  t<‘chnu  al  failure  for  each  subsystem. 
Katiiifts  were  ajytrejtated  to  produce  a  qualitative  estimate  of  program 
risk  for  the  system  as  a  whole.  As  planned,  the  siqiiHirt  contractor  tally 
dcK  umented  the  proci'ss  and  results  for  the  use  of  tlu'  program  offiiv. 

Uisk  efforts  for  I  b  ot  her  s>  stems  met  thn*e  or  four  of  our  fi\  <*  criteria, 
and  efforts  for  all  s\  stems  met  at  least  one.  Since  a  ti'chnical  risk  assess- 
im-nt.  as<lefine<l  here,  must  fulfill  all  fivi-  criteria.  have  not  |)ro\  ided 
a  detailed  description  of  efforts  that  did  not  fulfill  one  or  another  indi- 
\  idual  i  riti'i  ioii.  Hut  two  examples  w  ill  illust.  ate  what  wetyiiically 
found  For  the  .\ir  Force  lnier-Ser\  ice  Agency  .Vutomaied  Messajle 
I’rocessinjt  Fxchaufti' ( I  S  .\  ami’K)— a  hijthly  soiihisiicated  system  for 
command.  eonlr<i!.  commiinii  at mn.  and  intelhjtence-- program  mana);c- 
ment  staff  jtal  hered  inlormai  i<  on  technical  risk  t  hrou^h  various  ad 
hoc  n  etliods.  such  a.s  \endor  (  onfereiu  es  and  sur\'e\  s  (to  evalu.ite 
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di‘si>^n  ylten);iliv<-s)  and  reviews  l)y  cost  analysts,  users,  aiul  lahoi  atory 
r«‘pivs<*ntatives.  Itiit  this  input  did  not  fiK  iis  explicitly  on  te«  linical  risk, 
nor  was  it  dmuntentetl.  For  tlu'  .Army's  .vi.vi  tank,  three  ricv  K  analy  ses 
were  perfornUHl.  'I  he  analyses  priKliin-d  "hi^li.  "  "low."  and  "mosl 
lik(‘ly"  c«)sl  estimates  tor  each  of  the  Mi.M  s  siihs.\  stems,  hot  no  explana¬ 
tion  of  ti'chnical  risk  was  i)n*vided  for  any  one  subsystem  Staff  mem¬ 
bers  rejKniedly  considered  lechmeal  risk  when  providin}*  ntv  i;  input, 
blit  this  also  was  not  do<  nmenti  d. 


For  three  of  the  five  criteria,  we  found  differences  iH'twtsm  the  .services 
.Most  Army  and  Air  Force  effoils  were  prospcvtive.  but  a  majority  of 
those  in  the  Navy  were  not.  Funlu'r.  the  Army  usually  documented  its 
efforts;  the  Navy  and  Air  Forci*  usually  did  not.  Finally,  the  Army  and 
Navy  usually  rejnstted  their  efforts  within  acquisition  jihases;  the  Air 
Force  usujilly  did  not. 

LX-spite  thi*  hiUher  incidence  of  Army  effe.ils  meetin^t  thesi-  thris*  cri¬ 
teria.  pro)iram  manatfement  offi<  es  of  neither  tl.e  Army  nor  the  two 
other  ser\  ices,  in  tJie  majority,  met  all  five  criteria. 

In  summary,  projiram  offices  for  none  of  the  2"!  systems  iHu  formed  tlie 
sort  of  risk  effort  suntjested  by  Initiative  1 1 — a  quantitative  a.ssessment 
of  technical  inoblems  for  use  in  risk  budiietinjj.  I  'sinjJ  a  generic  concept 
based  on  five  essential  criteria,  we  found  only  .3  proi;r:un  offices  ix'i  - 
forming  efforts  that  met  all  five  criteria.  The  remaiiunn  addressed 
risk  in  some  way  but  did  not  fulfill  at  lea.st  one  of  the  criteria. 

We  reiterate  that  in  our  study  we  did  not  i-valuate  the  effect  of  risk 
I'fforts  on  i-ost  or  schedule  iirohlems.  lienee,  we  cannot  say  whether 
quantitative  asses.sments  or  risk  efforts  nuvtinft  all  five  criteria  have 
actually  lM‘l|H'd  reduce  cost  growth  or  time  delays.  Mut  risk  efforts  m 
most  jrrojjram  offices  cannot  h a \  i*  .served  well  a;;  technical  risk  assess¬ 
ments.  The  lack  o*'  diK  iimentation — diKumcntation  is  essential  for  the 
needs  of  decisionmakers — was  the  most  lotumon  tlaw. 


In  this  seition.  we  describe  how  the  program  manaKement  offii  es  con¬ 
ducted  their  risk  idforts.  reHardless  of  whether  t  hey  met  all  five  criteria 
for  technical  risk  a.ssessment.  From  our  ii-x  iew  of  research  on  organiza¬ 
tional  manaj’ement  and  ri-k  assessment,  we  concluded  that  imple- 
mentin^ta  risk  effort  entails  at  least  threi'  decisions,  For  each  dei  ision. 
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tlie  persons  implementing  the  effort  select  one  of  the  f)ptions  shown  in 
table  3.3. 


Table  3.?:  Oecitions  and  Options  in  the 
Implementation  of  Risk  Efforts 


Decision 


Option 


Format  lor  rating  risk  Narrative  description 

Qualitative  rating 
Quantitative  rating 

Scope  the  ratings  will  cover  All  subsystems 

Selected  subsystems 
Systeni  as  a  whole 

Input  collection  procedure  Single  rater 

Group  discussion 
Independent  raters 


There  are,  of  course,  implementation  decisions  other  than  what  forniat 
will  be  used  to  rate  risk,  what  scoik'  the  ratings  will  cover,  and  how  d:ita 
on  risk  will  be  collt'cted.  One  is  how  much  staff  time  will  be  devoted  to 
an  efft'it.  Hut  it  i.s  for  these*  three  decisions  that  particular  options  (sfa'c- 
ified  lH.‘low)are  most  likely  to  priHiiice  accurate  and  usc'ful  results. 

\'arious  program  circumstances  can  constrain  thr*  choice  of  implenicnia- 
tion  options.  For  instance,  the  decision  regarding  procedures  for  col¬ 
lecting  data  depemds  partly  on  the  time  and  staff  skills  available  for  this 
task.  Similarly,  the  deeision  (tn  rating  format  deix'nds  partly  on  the  coni- 
jilexity  and  nuuurity  of  the  system  being  develoin-d.  Aeeordingly,  for 
each  implementation  decision,  we  have  indicated  the  preferable  option 
and  rejMirt  the  number  of  rusk  efforts  for  which  this  option  was  s<‘lected. 
Hut  we  do  not  suggest  that  all  efforts  should  lx*  implemented  in  the  .same 
way,  and  we  have  not  ineluded  any  implementation  option  in  our  cri¬ 
teria  for  gauging  an  effort  s  quality. 


Ratinf5  Format.s  ^  Ihn-e  ojitions  for  deciding  how  to  r:ite  the  technical  risks  a.ss<Kiated 

with  a  system  are  narrativt*.  qualilati%<'.  tiiid  quantitative.  NaiTairte 
infoi-iraUcm  describes  [Kdcntial  problems  th;it  ma\'  preclude  reaching 
IK-rforniance  requirements.  sonH‘iiin''s  it  also  indii  aU's  the  source  of 
each  prolilcm  and  jMissilile  >oliitions  to  it  or  design  alternalivcs.  .An 
examiile  is  the  narrativ<' <:«  scrijilion  of  risk  associated  u  ilh  a  tomixmcnt 
of  the  .-Xrmy’s  .Advanced  Helicopter  Improvement  Program: 

"Until  (i  iinlrarliir  alln  iialivf-sj  lia’  c  llii«  u  a  MMs  fiiiasi  niimnlril  si>;lil  |  mi  Ua  ir  air 
I  raft  .  and  liavc  di’iiinuMrati  il  i  aiiui  '-  aiul  Malulit  \  i  niiip;.'  itilf  w  it  li  |■.\ 
ri'ninrcriu'iils  lUat  ii| .  -av  tw  ililtu  iill  in  niiiimi/.r  vtiffncss  amt  ui  ivtlil  Tlir 
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MMS  lould  impart  hi>;h  main  rotor  blado  tH-ndinc  loads.  Tho  main  rotor  blade  bal¬ 
ancing  traikinn  could  Ih-  diff'cull  .  .  ."  ( Kox.  ItrHl ,  p  A-IG ). 


Qualitative  estintates  for  the  likelihf)od  of  not  mectin}'  iK-rformancc 
requirements  are  usually  expressed  in  an  ordinal  rating — from  high  to 
mf)derate  to  Itrw — or  in  a  eodt“d  ordinal  rating — in  which,  for  example, 
red  is  equivalent  to  high,  yellow  to  mtKlerate,  and  grcsm  to  tow,  I  he 
Navy's  Joint  Tactical  Information  Distribution  System,  for  example, 
coded  eiLsily  solved  problems  green,  ixissible  major  problems  yellow,  and 
any  major  problems  that  seemc'd  potential  "show  stoppers"  red. 

Quantitative  estimates  of  risk  use  a  fraction  expres.sed  as  a  decimal  to 
represc'ut  the  probability  of  meeting  or  not  meeting  jx-rformance 
requirements.  One  instance  of  this  is  in  the  effort  for  the  Air  Force  Anti- 
satellite  Weapon.  The  program  office  rated  the  probability  of  success  for 
each  AS.AT  subsystem  and  then  aggregated  the  figures  to  produce  an 
overall  probability  of  succe.ss. 

Narrative  ratings  have  the  advantage  of  content;  they  describe  the 
txitential  problem,  itsscnirces,  and  its  |x>ssiblc  solutions.  Hut  the  narra¬ 
tive  alone  does  not  indicate  how  raters  would  estimate  ihe  level  or  mag¬ 
nitude  of  risk.  Qualitative  and  quantitative  c*stimates  do  indicate  levels 
of  risk.  Sueh  estimates  alone,  however,  lack  the  content  firovidt'd  by 
narrative  descriptions.  Systems  that  are  well  into  development  or  not 
ver>-  complex  might  not  rt^quire  both  a  discussion  of  risk  elements  ami  a 
spc'cification  of  risk  levels.  Hut.  in  general,  the  most  informative  format 
would  combine  narrative  information  with  either  qualitative  or  quanti¬ 
tative  ratings. 

Only  narrative  ratings  were  usc-d  for  r>  systems  {20  pc'reent)  in  our 
study.  Discus-sions  of  risk  in  the  Navy's  Trident  II  program  office,  for 
instance,  focused  on  the  engiiu'ering  asiXH  ts  of  tc'chnical  problems  h' t 
did  not  ordinarily  entail  qualitative  or  quantitative  ratings.  Kiftt'en  sys¬ 
tems  (00  [KTcent )  wcTc  rated  for  risk  in  qualitative  terms  without  narra¬ 
tive  details.  Three  systems!  12  percent)  wen*  given  quantitative  ratings 
without  narrative sujqxirt. 

A  narrative  w;ls  combined  with  qualitative*  or  quantitative  ratings  or 
iMith  for  I  systems  f  10  percent ).  For  the  .Army's  .Min*.  narratives  for  sub¬ 
system  risk,  like  the  narrativ  i*  quoted  above  for  the  ma.st-mounO'd  sight, 
were  aciompanied  by  onlinal  ratings. 
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to  use  i)ii;ililaliv(“  ratings  only:  tlic  Air  Fon c  provided  either  narnitive 
inlorniatioii  or  «)ualitative  raliiins.  Tlie  Army  iisiiall\’  rated  risk  in  tjuan 
tilati\e  terms,  tail  it  ('omlmx'd  (jiiaiUitati\'e  with  qualitative  terms  in 
OIK'  risk  etfort  and  with  qualitative  and  narrative  information  in  two 
efforts.  'I’he  Army's  ttreater  rehaiue  on  qiiantilativi'  rations  is  not 
simply  an  artifai  t  of  the  TRV  K  anal.vses  it  used  to  Inidnel  for  risk  hut 
can  he  aeeoiinted  for  hy  effoiis otlu'r  than  TICVK.  However,  llu'  Army's 
familiarity  with  Tltv'K  may  lu'l[)  evplain  its  more  fre<iueni  usi'  of  (plant i 
tative  t(  ehnieal  risk  ratinns. 


Kfforls  to  assess  risk  nia.v  focus  on  a  .system  as  a  whole  or  on  siihsj  s- 
tems  such  as  hardware  eoinjionents  or  software  suhroutines.  All  siihsys- 
terns  may  he  ass«'s.sed  for  risk  or  only  those  for  whir  h  there  seems  to  Ix' 
some  uncertainly  renardiiin  [lerlormance.  Kvcept  jK'rhaps  for  systems 
that  are  relatively  mature  or  simple,  an  eflr-rt  coverinn  all  subsystems  is 
likely  to  he  more  useful  than  one  coverinn  only  the  system  as  a  whole  or 
only  some  of  its  subsystems.  Attention  to  the  system  its  a  whole  may 
prodiK  <•  an  accurate  ('stimaie  of  overall  risk  hut  will  not  by  itself  iden- 
tily  tlie  more  prohlenialic  sulisystems.  Similarly,  an  effort  ineor|xiratinn 
onl,v  .select('<l  subsystems  will  not  pnaliice  tin  estimati'  of  risk  oserall, 
and  it  may  not  uh'iitify  the  subsystems  that  were  not  selected  or  re()ort 
the  reasons  for  the  selection. 

For  '2  systems  (H  jx-rcent ) efforts  were  condm ted  only  for  the  system  ns 
a  whole.  For  1 1  (H  |K'rcent )  risk  was  appari'iuly  rail'd  for  s<'le(  t('d  sub¬ 
systems;  for  10  (40  iHTcenl ),  it  was  rated  for  all  subsystems. 

Differeiu  es  emerned  in  seo|x‘.  The  Navy  usually  covered  some  but  not 
all  siibsysK'ms:  the  .Air  Force  most  often  covered  all.  In  no  ca.se  did  the 
.Army  jjear  an  effort  to  a  sysK'in  as  a  wholi'  but  instead  ass('ss<'d  all  and 
seU'cted  subsystems. 


The  jiriK  edures  that  are  used  to  collect  data  can  affect  the  lomprcl'.en- 
siveiiess  and  completeness  of  the  in|)Ut  lo  an  ass(",snu'nl  as  well  as  tiu' 
\alidil.v  ol  the  consi'ipient  output.  One  jx'ison  m.iy  comiM  iently  identify 
and  rate  risks  Hut  if  tune  and  ri'sourei's  ]x'rinii .  si-veral  raters  workinji 
as  a  )rroii(>.  eai  h  with  particular  expi'i  ii'm  i'  a. id  areas  of  exiK'i  l ise.  are 
more  liki'h'  to  prodiiee  mori'  accurate  injiut .  i's|H'ciall\’  it  the  r, iters' 
assumptions  are  s(x'll<'d  oi;i  and  technical  dei.nls  is  well  as  possible 
solutions  are  (irovided.  Communication  ;,mon>t  rati'fs  i  an  >;enerate  new 
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views.  Input  can  be  c(»llect<‘(l  in  a  stal  l'  (liscnssiim  i»r  teclmical  issues  or 
in  a  survey  (iisiii)'  itr  -rviews  or  written  ()uestio:'.nair<‘s) 

In  a  survi-y.  injait  is  tollecttal  *rnin  indejxMalent  raters  ami  then  tabu- 
lati'd.  and  dis<-repaiu  ies  are  n'solvt'd.  'I'lie  advantage  of  in|)Ut  from  s<-v- 
eral  raters  workin>f  indeiM-mlently  over  ^roup  discussion  is  that  iiroup 
pressures  and  tune  t’onstraiiUs  do  not  pn-maturely  close  issues  risnnrin>t 
exte  nded  attention.  Thus,  for  all  but  thi'  most  mature  and  least  cor  iple.v 
systems,  input  from  sc'Veral  imleixmdent  ratiTs  is  preferable. 

Ibsk  efiorts  for  :i  of  our  systems  ( 1 2  ix-rcimt )  relierl  on  one  jK-rson  to 
liandk'  the  effotl.  .Anotlier  17  (tiS  jHTcent )  colicctcri  data  from  two  or 
more  raters,  and  14  of  these  held  at  least  one  mei‘tin>j  at  whic  h  technical 
risk  was  discussed.  Five  (  id  ixTc  cmt )  collec  tisi  input  from  indeinmclent 
raters.  ('I’wo  usc-d  Ixiih  staff  di.sciission  and  indei>endent  raters.  1 


Program  Circumstances  and 
Implementation 


For  format,  scojk*.  and  input  prcK-cdurc'.  wc'  have  cited  t  In-  options  most 
likely  to  ifenerate  usedul  information  on  risk.  \V(>  also  found  t  hat  few  of 
the'  pronram  manau«*ment  office's  sc'lected  thc'se  oi>tions  when  they 
implemented  their  effons.  Hut.  as  we  noted  almve.  t  ire umstam  c  s  such 
as  available  staff  time  and  a  system's  compli'xity  can  affect  impli'incnta- 
(ion  decisions.  Wc*  did  not  attc*nipt  to  rate*  sue  h  c  ircumstances,  sim  i*  their 
measurement  would  Im*  lu^thly  siibjec  tiv  c*  and  impr  isc*.  Theri'fore.  we 
cannot  In*  c  c'lta;:;  that  the  implementation  dec  isn.iis  of  aiu  of  the  ollicc's 
were  either  apjiropriatc* or  inai>propriale. 


At  the  very  least.  howc*vc*r.  a  protfram  office*  should  consider  its 
system's  complc*xily  and  maturity  w  In  n  makin>;  thc*se  dc*c  isions.  If 
implcmc*ntal  ion  options  arc*  selecli*d  solc*ly  in  rc*s|)onse  to  staff  availa- 
bilit.v  and  olhc*r  constraints  not  s|H*cific  to  a  systc-m  itsc*lf.  one*  cani;ol  lx* 
confid(*nt  that  the*  rc*sults  will  lx*  the*  most  useful  (xissiblc*  m  the  furthc*r 
development  of  the*  sy.stc*m.  .\n  office*  handling  a  complex  nc*w  sN  slc-m. 
for  e*xam|)l<*.  should  at  least  consider  performing  eiuant native*  analv  se**; 
in  wh'ch  the  risks  assex  i;itc*d  vv  ith  all  the  subsvsiems  can  lx*  pre'e  iscly 
aiyfre^tated  It  i*'*c.!.;  more*  appropriate  for  another  office.  mana>;m>; 
enhancemi*nts  to  an  existnijt  sysie*m.  for  example,  to  rcsiuin*  onlv  a  brief 
desc  ripl  ion  or  e|ualitativ c  rating  for  eac  h  <*nhanc  i*ment 


In  our  studv.  responck'nis  i  il<*d  a  wide*  ranite  of  reasons  undi'i  l\  iiiu  the* 
implementation  of  their  task  edforls  .Arnon^;  those*  most  freijuenllv  e  iled 
were  slat  f  ex|«*ricncc  vv  ii  h  similar  efforts,  confide'iicc*  u.  the  rc'sults,  and 
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int5K»s*-<l  at  liiitluT  <'<iiiu'iaiuJ  Irvcls.  In  only  (>  ot'l'ici'.s  ('.M 
l^'rci'iit )  (lid  a  ri's}K»n<lfm  say  that  features  of  the  systi’in  iis<‘lf  wvix 
considorocl. 


Summary  of 
Implementatioi) 


Sina*  ini|)lcnicntation  options  dojK'nd  partly  on  proytram  circumstatuos. 
wo  havo  not  attoinptod  to  s|H-cify  any  ossonlial  oritoria  for  iini)lomcnia- 
tion.  Ilut  a|)parontly  tow  offices  oonsidorod  uroyjram  circumstances 
when  Itioy  im|ili-nu-nted  their  risk  «‘fforts.  and  few  offices  soleitcrl  the 
options  Itiai  are  in  fjenoral  most  likely  to  jinKluco  accurate  and  useful 
results 


What  Infomialion 
Technical  Risk  Is 
Available  to 
Decisionmakers  in 
Review  Process? 


on 


the 


I'o  answer  the  question  on  th.e  availability  of  information  on  technical 
risk  for  those  who  make  ftecisions  at  the  milestone  reviews,  we  re\  iewi’d 
d(K  uments  and  hriefiii)’  materials  ( minutes  and  scrii>ts  as  well  as  charts) 
tireparisi  by  jiro^tram  management  offices  to  dcsciihi’  their  l<‘<  hnical 
prohleiiis  and  jilans.  As  we  noted  in  i  hajiter  1.  the  din  uments  reijuired 
at  milestone  revii'ws  includi*  the  system  concept  paiH'r.  decision  coordi¬ 
nating  paiH'r,  test  and  evaluation  master  plan,  inii'p'ati'd  i'ro;;ram  sum¬ 
mary.  and  a  paiK-r  on  acquisition  strateiiy.  As  we  disciisM-d  in  di  i.iil  in 
ehafiier  2.  ixip  re>jiilaiions  sjM'cify  that  each  diKiiment  must  include 
information  on  the  tis  hnical  risks  (xised  by  a  system  or  the  progress  of 
risk  reduction. 


We  reijiK-sied  offuial  i-opies  of  these  (iis  uments  by  name  and  briefiu); 
mati'fials  by  nuU'stom-.  We  also  nspiesied  other  teihmcal  doi  umi'iits 
that  weri‘  available  to  ri'vii'wers.  such  as  mission  element  lus'd  state¬ 
ments.  pro)train  manaHi'ment  dircsti\es.  and  technical  advisory  pan<‘l 
reiKirts.  Some  <1ik  uments  wen-  mi-.sinjf  from  a  few  offices,  es|K'(  iallv  for 
systems  that  had  skipped  or  not  yet  n-ached  a  milestone  and  that  had 
pasvd  a  milestone  In-fore  the  re»iuiremi-nt  for  sjiecific  diK  uments  had 
iM-en  established  (Hfier  diK  uments  were  av  ailable  but  evcluded  from  the 
analy  sis  if  the  relevant  milestone  d.ile  could  not  Im-  |iin|)ointed  or  the 
milestone  had  tx-en  skij>|K-d  or  not  yt  ri-ached  When  we  were  jirov  nled 
with  sev  eral  v  ersions  of  one  ilociiment  I  for  example,  an  ordinal  for 
milestone  1  and  its  later  u|Klai<‘).  we  included  each  v  ersion  in  lh<-  anal¬ 
ysis.  /Ncross  all  1  he  o((ices.  we  examined  L’H  milestone  dot  uments  and  17 
sets  of  briefnij;  materials. 
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Tabla  3.4;  Th«  Sourcat  and  Typat  of 
Information  on  Tachnical  Risk  at 
Milaatonea  I  and  II 


•Most  iniU'stune diM'iiim'nt.s  incliuiod  ,nf<iriii;iti(iii  um  i(‘i'hnu';il  i  isk:  HU 
IXTei'Hl  at  milostoiH'  i  and  7(i  ivrtr'nt  at  milcstoiii-  II.  Althminli  iKindm 
not  rntx'i  it.s  ntiuiri'iiifitt  for  (oi  hnital  risk  information  in  all  iIicm'  diK  U- 
inonts,  for  oac  h  sj  stotn  at  each  milostont*  IIior*  was  at  least  one  diK  u- 
ment  providinn  sueh  inOirmation.  fintlier  analysis  indieatc'd,  however, 
that  the  information  on  risk  was  ii.ailtsjiiate.  In  almost  all  the  din  nments 
(none  of  whu  h  had  quantitative  ratings),  the  information  was  a  nerra- 
tive  or  a  quahtativ  ratin);  of  risk  for  the  system  or  siihsystems.  Few 
diK’iiments  siK'<-ifie<i  an  eifort's  seofx'  or  analytical  a[>|>roach  at  either 
milestone.  (Stv  table  :}.4.) 


C31V 

Oocumenl 

t 

Contains 

technical 

risk 

Information 

Gives 
technical 
risk  rating 

Cites 

approach 

Cttftt 

fcope 

Test  and  Craiuj.  .  -  maaic;  p'an 

2b% 

25% 

0 

0 

SjSiefn  cooceo!  oaper 

1(X) 

'00 

0 

0 

Acquisition  st'dtcgy  papcf* 

too 

irjO 

0 

20% 

All  docum«nts 

B0*» 

80% 

0 

7% 

Mil«tton«  II 

Test  and  evaluation  mastc-t  plan 

60% 

hO'o 

10% 

10% 

Decision  coo'Oinaling  paper 

60 

80 

0 

0 

Integrated  program  summary 

00 

50 

0 

c 

Acquisition  strategy  paper' 

too 

too 

0 

0 

All  documents 

76% 

72% 

3% 

3% 

*'nciijOcs  OOCuff'ef'tS  cnfiiied  AcMutvl-oo  Plan 


for  example,  the  ti’st  an«l  evaluation  master  plan  is  supimsed  to  list  erit- 
ieal  issm-s  to  Ik’  ri-solverl  In  testinfi — issues  arising;  Irom  oj^'iational 
requireimaits  and  from  ttH-hnical  risk.  When  a  jilan  laeks  a  deseription  of 
the  risk  effort  or  ratinjjs of  th<'  risk  ;<ss<k  lated  with  <  riiK  al  issm-i, 
readers  may  know  what  is.siies  are  etinsitieretl  eritii  al  hut  will  not  know 
(or '.  an  onl\'  inlerithe  level  of  leehnieal  risk  assoeialed  with  each  issm* 
or  t he  (piality  of  the  risl<<>ffort  for  that  system.  For  milestone  I,  T.'i  |h-i  - 
cent  of  the  plans  lacked  e\|i|icil  risk  information  of  this  sort;  for  mile¬ 
stone  II.  -lU  js'i’cent  lacked  it. 

In  another  anal>  sis,  we  examined  all  the  diK  iimeiils  the  offices  prm  u'eil 
iis.  m<  Imimi;  dm  unients  not  reqiiireil  for  milestones  and  .lociimenls  lor 
n  liK  h  no  milestone  dale  could  he  pin|Miinied  'I'he  p.ittern  o]  results  fur 
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<l«Kiiii\ciit s  oMTiill  (iiijili*  .ill's  i!ic  jiatli'rn  \m'  ruiiiu!  Im  iiiil\  ilic  iiulr- 

'-....u'  ilm  umi'iUs  aiiii>ii)4  til*'  t  -i.il  til  1  !t!  i!u,  iiiiii'nis.  ti  1  jK-n  fiii  pi  n 

Miird  risk  iM(tirni.ini>ii.  iisiiallv  raiiiik''  I'M  rari'l\  a  (’I'-i  iijii mn  nl  s(  i>i>.' 
or  apjiniai  li. 

W'c  cxaiiititi'd  iiiiiiiiti's  ;in.l  si  ripts  I'or  l>ni'liii;^s  in  ordi  r  to  di'U'i  ininc 
\\liat  virt  oT  111  liOK'.il  iisk  iidorinatioii  \\as  prov  idi  d  lo  H'\  u-wits 
orally.  In  most  i-avs.  this  inrorinaiion  look  l lie  rorni  o|' i  harts  shou 
risk  ralin;;s  for  snlisysirnis  or  a  sysioni  o\  rrall  Kor  a  IHS 1  iis.sia  ri'\  n-w 
of  I  hi*  Na\  y  I  I'tTS.  for  tnsianri*.  hnofin^  rharis  pro\  idcd  a  (|ti.ihiai  i\ c 
risk  ralinn  ( low  to  inoili  raii*)  for  liic  sysii  in  as  :i  whotr.  <  hails  used  lor 
a  I!iH  l  rovicw  of  tin*  Air  Fom*  M.irk  XV  IKKroinhal  idonlifiratioii  s\sii  in 
did  not  contain  (|italitattvi*  risk  ralinks  hut  did  di*scrihi‘  sonn cs  of  in  h- 
nical  risk  and  design  approaches  for  various  snhsysteins.  At  niilesioiu*  I. 
•Id  pi  rcent  of  the  hriefinn  inalertals  pro\  ided  technical  risk  informal i;.;-.. 
which  consistt*!!  of  risk  ralinjls.  None  cited  the  sco|h'  or  approach  of  an 
effoil .  Al  milestone  II.  dO  |n*ri  ent  cited  risk  ra!in;t.'-.  ...id  only  rarety 
wen  s(oiH‘  and  approach  cited 

In  sumniai'v.  tKiP  rej;iilalioi*\  nsiiiirt*  that  all  niilestone  iIik  niuents 
i.'u  liide  inforniaHon  on  techim  al  risk  *ir  risk  reduction,  .Most  of  the  docii- 
iiienis  We  revicweil  for  this  study  iiu  liide*l  siicli  infoi  iiiatioii.  hiif  some 
did  not.  More  imjwiiiatilly.  the  risk  iinorinat ion  that  was  a\ail.il'le  in 
I h<'se  d(K  nmenis  ran'ly  inilicatiHl  the  sci»ik*  of  the  efiori  or  t  t.c  i  • 
nal  afiproai  h — two  if**ms i  ritiial  toa  lhoroii^;h  e\ aiiialion  o*  'he  lei  h- 
meal  risks  iioseil  h\'  a  system  The  hriefinn  materials  we  ••vaim.  ed 
stifUlesl  that  risk  inJorniaiion  wasoften  not  proviih'd  orall>.  alihoimh  it 
is  [hissihle  that  r*'\  lowers  r.iisi'd  tjiieslions  alMiiii  risk  al  tin*  hriclmi’.s 
Tlu*  inhirmat  mn  kfetierally  pro\  id»'d  was  a  raiinif  of  iis  him  al  risk,  hut  as 
with  s>st«*ni  <1<M  unK'iils.  hriefint;  nr.itertals  rarely  sjks  died  the  sco|k*  of 
the  ratiii).;  and  the  aiialv  lical  apjiroat  h  that  prisliK  ed  it 

ScofX*  and  Format  j  t'*'  format  lor  risk  r.ittnrts  nnnis  dose  atleniion  hecaiis**  \er\’  tew  d***  - 

iinients  prov  ide  risk  mioi iiialioti  other  than  ratings  and  hecaiise.  as  we 
noted  eartier  in  tins  «  h.i(.ter.  ih<*  most  iis<'liil  lorniat  would  * umlinie  a 
narrative  description  of  lechni*  al  prohlems  with  a  ipiahtat iv **  or  (|iiami 
tative  rat  in>; 

For  this  part  ot  liie  aiiaiv  sis.  we  expanded  onr  concept  of  sciijk'  hi  the 
prc<  ediiiU  discussion,  we  tin  iisetl  on  whet  tier  milestone  di M  l miciits  and 
hnel  in>;  mai  el  nils  <  |Ie<l  l  h*'  scoJm-  oI  the  risk  et  tort  all  or  sele*  ted  suh 
sv sti-nis  or  ( In' sv  stem  as  a  vv  hoi*'  \V(  loiind  that  viTV  li'w  did.  alt hou^;h 
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)<«  3.5:  Th«  Fonnet  and  Scopa  o( 

Jk  Ratinga  m  Miieaiona  Oocumarts 
i  Bnafing  Chads 


.imniary 


I'isk  wrrt'  rr|Kiric«f  in  tfic  majority  of  availaltlo  (iiM  imtcnis  and  m 

anoni  halt  of  tlu“  availahU-  l>riolin>;  inalrnals,  'rims.  f\t‘n  if  the  m  o|h'  of 
tliiM'lfoil  ua'Niiot  ntftl.  ui‘ laii  still  ask:  W’hal  .is  i he  s<'oj«- ol  Iho 
nilinj^sV  \V('  noii-<l  »%irlicr  that  risk  4•Iforls  arc  Hcncrallv  more  useful 
when  they  eo\ cr  all  subsystems,  not  jiist  .s<‘le»  ted  siihsystems  or  t he 
syst'-m  as  a  whole,  (’omhininjj  lhes«‘  two  concerns,  we  crossa  he  sified 
milestone  dix  iiments  and  I'riefinj;  materials  by  their  format  and  sc  ojk*  to 
MH*  in  moH‘  detail  how  risk  was  ratisl  lor  mileston.  rev  lew  s 

Few  diH  iiments  ( 1  a  jxTcent )  provided  descriptions  of  its  hnical  problems 
alonjl  w  ith  a  risk  ratin}*.  and  tin*  tliK  iinn-nts  that  did  coveretl  the  system 
as  a  whole  or  se'eetisi  siibsyslc*ms.  None  covensl  all  siibsystetns.  Of  the 
briefinj'  materials.  2il  ix-nent  provnltsl  d»-seriptions  alonn  with  (pialita- 
live  ratinHs  and  Icilfof  lhes«‘(  Itt  in-rcent )  tovensl  allsiibsysletns.iSee 
table  :i..'>. ) 


Formal 

Scopo 

Daacriptiva 

Ouatitativo 

Both 

F^iletlon«  documtntt 

System  as  a  ^ngic 

7s, 

Sv'ieciod  SNhsysicms 

4t 

'.2 

0 

All  siihsyslomi 

0 

2 

0 

Bdefing  charts 

System  as  a  wiioio 

0 

40 

0 

*■  ek'r.tc.sJ  S'lps, stems 

?0 

20 

to 

AM  S.ihSyS'oms 

0 

0 

tf) 

No  prov'ram  manajtement  olfne  has  qtianlified  and  !iiid>tete<l  for  l«s  h- 
nical  risks  as «  alhsl  lor  by  Initiative  1 ! .  Althouith  the  jirojiram  oflu-es 
for  all  2o  systems  bava*  made  .in  e,fort  to  identifv  their  tec  linic.d  risks, 
only  d  eondiictcsl  risk  efforts  that  misd  our  criteria  for  tec  hnii  al  risk 
.'-•..sesstuetil . 

In  addition,  we  found  wide"  variation  in  how  risk  efforts  were  implc-- 
menlc’d  The  im|i|enientation  of  an  eflort  deiieiios  partlv  on  proijram  cir- 
i  iimstances.  so  we  <  annot  e\|>cs  i  all  c'fforts  to  liave  Iks-ii  c  arricsl  out  in 
ev.ielly  the  s.ime  w.iy  a.nd  «  annot  Ik*  certain  that  those'  we  evaiiiimsl 
renecii'd  inappropriate  implemental ton  di'i  Nions.  lint  most  ol  the 
offices  did  not  consider  the  comj'IeMty  or  m.nurity  of  their  svstems 
w  hen  c  hoosim;  implementation  options  reit.irdink  format.  sn>|K'.  and 
data  collec  lion  jircKedures  Therefore'.  |i  Is  not  likely  that  I  he  ei'oirts 
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they  implcmontod  w«*ro  :ls  it'i  they  nniUl  have  liir !  !»•  fur¬ 

ther  development  of  their  systents. 

The  ser\  let's  and  the  i>sakc  must  make  dit  isions  reRardin^  the  I'ai  e  and 
direction  of  these  programs  durinj*  mih'sione  reviews.  Most  milestone 
dotniments  providtsJ  some  information  on  tis  hnical  r:sk.  hut  this  infor¬ 
mation  rarely  combined  narrative  information  with  rating.s  for  all  sut>- 
systems.  Our  analysis  of  briefing  materiaK  suggest.,  that  the  (irogram 
mtinagcment  offict's  w<*rc  iinliJ;*‘lv  to  add  further  details  orally  ( inly 
almut  half  of  all  siu  h  materials  cUetl  lis  hnieal  risk,  and  the  materials 
that  did  rarely  combiiieil  narrative  information  with  iisk  ratings  and 
rarely  csivercrl  all  subsystems. 


Pair  &2 


H#V  !i  It^chnlrai  Ki*li 


Difficulties  with  Current  Approaches  to  the 
Assessment  of  Technical  Risk 


\\  hen  wo  collect od  oiir  data,  some  issues  arose  that  are  not  k)\  ered  in 
our  study's  six  initial  evaluation  questions.  In  this  chapter,  ue  discuss 
these  issues.  They  concern  jiroKram  offices'  working  cefinit ions  of  tech¬ 
nical  risk  and  risk  rating  categories,  the  provision  of  r.sk  information  to 
decisionmakers,  ixiu's  training  in  ‘echnical  risk  a.s.sessment.  and  the  risk 
information  contractors  jiroviut  to  program  offices. 

Definitions  of  Risk  and 
Risk  Rating  Categories 


If  definitions  or  rating  formats  are  inconsistent,  the  disisionmakers  will 
nwd  to  ask  for  elarifieation,  and  this  could  take  considerahle  time  For 
example,  if  .sub.sy.stem  risks  are  not  rated  in  terms  i  it  are  familiar  to 
reviewers,  program  staff  may  be  required  to  revise  the  ratings  or  con¬ 
duct  an  (mtirely  ni'w  asse^'inent.  Worse*  yet  is  that  inconsistencies  may 
never  be  ree<ignized  and  t  .q  program  office  managers  (tliat  is,  thi*  i  hii'f 
engineer,  contract  officer  .;tui  program  manager)  may  base  daily  dec  i- 
sions  on  teehnieal  inform;  >n  tiiat  is  vague  and  quite  jMissibly  liiis- 
k>ading.  '/  his  would  also  .  .  ;  reviewers  at  higher  levels  in  the  .vn  ices 

and  the  tiSMcc.  where  nu.  .-ahead"  decisions  are  made. 


It  is  imjxirtant  that  technical  risk  be*  tlc-arly  and  consistently  concc‘ptual- 
lic'd  within  and  across  the  iiri'gram  management  offices.  It  is  also  imjxir- 
tanl  that  r.sk  rating  categories  be  consistently  defined.  This  is  lruc‘ 
regardless  of  the  rating  formal— narrative,  qualitative.,  or  quantitatiie. 
Not  all  program  oflices  need  us(.  the  same  format.  But  it  is  nc'cessary 
that  all  those  that  use  a  qualitative  formal,  for  oxamiile,  define  higli, 
moderate,  and  low  risk  in  a  similar  wav. 


Dofinitio'ts  of  Technical  'W*  found  that  only  .") ; 

Ri.sk  nit  ion  of  tec  hnic  al  ri-.i. 

sta.T  members.  .\forc‘ov 
or  scTS  ic  (.  definiiiijns  o; 
deliniiions  are  ambiguo;,'  i-  . 
(uily  one  .Air  Force  offie.'  were 
definc's  risk 


:mgement  offievs  had  a  standard  ciefi- 
'•  [Kilicy  and  known  and  ain.lied  b\  all 
Iciits  in  only  .'j  ol fiec's  c  iic‘d  eil Iter  (iciii 
i  IS'.  (fK'rhajis  in  part  because  these 

■  isc'Ussc.d  in  c  hapter  2)  Kesixindents  m 

■  irc*  that  Air  Force  Ki'giilation  Tn-l.A 


If  neither  doeuir.enied  : 
have  estahlishc-d  a  star 
rc-sjMindmUs  actually  i; 
Table*  1. 1  summari/.e-^ : 


mns  nor  pi  n. m  management  (Kdic  ies 
iimiuni  (if  ri-.  ,at  clefinilicns  did  the 
iiidard  or  i  ..t .  e  ■  n-  day-lo-dav  wurk'.’ 

nswer. 


Pi(Cr  54 


CiAO  'r.MOKft  5  Tfrhnlcftl  Hi.'k 


Table  4.1:  The  Definitions  of  Technical 
Risk  Used  in  the  Program  Management 
Offices 


IX'finitioiis  of  Risk  Rating 
Categories 


<  'hiipirr  t 

hifrirtillUM*  with  i'urri'iil  .\|>|iri»«rlirH  lo  Itir 
A-*»N**sNiiit'nl  <»f  IWhiiirjil  Ki>k 


Number  of 


Definition  offices 

Lii'ciinood  o!  prcrjic-ms  can  be  caicuio'od  3 

Pfobabiiity  ol  faiiu'c  can  be  calculated  3 

Probability  o'  failure  can  be  calculated  (ji.en  schedule  or  cost  iim.ts  2 

Technology  is  unproven  or  beyond  the  stale  of  the  art  2 

Technical  nsi-  'S  loo  sub/ect/ve  to  define  2 

Probability  ot  taiiure  ana  tne  consequences  can  be  calculated  0 

Oflices  giving  inconsistent  definitions  4 

Ollices  C'wng  no  definition  9 


We  eiit«Tf<l  (tefinition.s  in  the  talile  if  all  ur  innst  resixindents  pnivided 
tiie  same  dennition  or  if  doi  iimentation  prov  ided  one.  No  office  cited  the 
IK'fense  Systems  Management  College  definition  that  was  Lased  on  the 
prottahiiity  and  consequences  of  failure  (quoted  in  cliaptt'r  2).  although 
5  offices  based  their  defimiions  on  the  itrohability.  but  not  the  conse¬ 
quences.  of  failure.  In  2  of  these  '>.  resiMindents  defined  technical  risk  as 
the  probability  of  failure,  given  limited  time  or  limited  funding.  In  3 
more  offices.  resiKindents  defined  technical  risk  as  the  probabilitc’  of 
failurt*  hut  did  not  cite  sclfilule  or  cost  limits. 

Other  offices  offered  definitions  that  wt're  similar  to  these  but  not  bitsed 
e.xiilicilly  on  the  jirobability  of  failure,  I'wt'  offices  based  their  defini¬ 
tions  on  the  degree  to  which  the  requirt'd  technology  was  unproven  or 
Ix'ytind  the  statt'  of  the  art  ( not  yet  evi'ii  partially  deicloiied ).  .And  3 
offices  defined  risk  as  the  exislemt'  of  a  technical  problem,  or  the  likeli- 
litMid  that  oiK'  would  arise,  but  not  necessarily  a  problem  that  would 
cause  program  failure. 


In  ‘1  offices,  the  definitions  we  wt're  givt'ti  wi're  inconsisti'iil  in  ways 
that  coultl  not  1m'  resolvt'd  by  taking  a  di'fmition  from  the  majority  of 
the  rt'sjMindents  tir  from  their  program  documt'ulation.  In  2  other 
tilTices.  the  ma.iorily  ol  t  hi'  rt'siMtntients  said  simiilv  that  tei  hmcal  risk  is 
tiKi  subu'ctivt'  to  define,  t  luformalion  was  not  suflu  leitl  for  coding  the  !* 
ol  lll'l'  t)l  I  It  !'■»  ) 


.\s  wf  noted  in  <  haptt'f  3.  most  program  mauagemenl  olfices  rated  risk 
in  (tuaiiialive  ii'rms — high,  motleralt'.  anil  low  tor  red.  yellow  .  and 
grt'i'H  )  In  our  inlt'i  v  lew  s.  w  <'  asked  resiKiudeiiis  how  i  be\  ilcfined  llit'se 
(|uahl.iti\(-  terms.  Tht'ir  answers  were  not  coiisistcni 
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St'von  offkrs  dt'fiiud  qualitative  rating's  in  narrative  terms  For 
example,  hi^li  risk  vv;ls  sometimes  defined  'ts  ''solvat)l<‘  if  the  si  hedule  or 
jx'i  fonnanee  ri-qinrements  are  ehanfjid."  moderate  risk  as  ".soh  ahie 
with  noeliannes"  (or  solvable  withoin  redueinf'  the  iK'rfornuime 
n-qiiiremettts).  and  low  risk  as  "no  problem."  Three  offi<  e‘  defined  qual¬ 
itative  ratinjis  by  a-ssljinin):  probability  ranges.  For  example,  an  HO-ikt- 
eent  ehaneeof  not  meeting'  jH'rformanee  requirements  was  liinh  risk,  a 
ehane<‘  of  21-79  jaueent  was  moderate,  and  a  chance  lower  than  21  la-r- 
eent  w;ls  low.  Two  other  offiics  used  both  narrative  and  quantitative 
terms. 


In  2  offices,  resmmdents  did  not  ajtree  on  what  ratinU  format  liad  been 
ii.M'd.  and  the  ineonsistem  y  could  not  lx'  resolved  by  taking  the 
ma.iority  s  answer  or  referrinjj  to  pr<‘>;ram  diK  uments.  Five  offices  used 
qualitative  ratinns  but  said  the  terms  are  tiKi  subjective  to  define. 

Neither  narrative  nor  quantitative  terms  are  necessarily  preferable  for 
defining  qiialitativi'  ratinjjs.  lienee,  this  variation  amon;;  the  offices  is 
not  a  problem.  Hut  when  wi*  examined  the  meanings  attributed  to  narra¬ 
tive  and  quantitative  terms,  we  found  inconsistency  [lersisting  liotli 
within  and  across  the  offices. 


Narrative  'I'ornis  for 
Dofiiiiiig  Qualitative 
Ratings 


Uespondents  provided  several  versions  of  narrative  terms  for  thei!'  qual¬ 
itative  ratings.  In  some  cas<'s.  resjKuidents  in  one  office  iirovided  more 
than  one  narrative  dctinilioii  for  high,  nuxlvnitc.  or  low  risk.  .Some  itefi- 
nitions  were  merely  distinctive;  others  were  contradictory.  Table  1.2 
suinmari/es  them 


Failr  'lA 
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I'hiipl^r  4 

fK/flruJtirs  with  K'ufrrfit  Appn>*rh<^  to  the 
Aiaeemuf  nt  ^TtH-KiticAl  Kink 


Table  4.2:  Qualitative  Risk  Ratings  in  ■BBBBBMBBBHmiHBBHBflHMOBHBIHBflKiHBHBflBHHnBHBBSKBBHI 

Narrative  Terms  Used  in  the  Program  Number  of 

Management  Office#  Rating  snd  term  _  office# 

High  (red) 

Solvable  with  changes  in  schedule  or  petfoin  ince  spocilicalions  6 

Bex’ond  the  stale  ol  the  ail  4 

Probable  (aiiure  *  2 

Maiof  pioblem  1 

Test  oian  not  yet  devised  ' 

Current  stale  ol  the  art  1 

No  definition  obtained  Irom  ollice  10 

Moderate  (yellow) 

Some  development  success  but  si'!!  uncertain  6 

Solvab'e  •'■ith  no  changes  ir,  ichcduie  or  soeC'iica'ioiis  2 

Test  plan  devised  but  testing  not  yet  completed  2 

Caution  2 

Beyond  the  Stale  of  the  an  1 

Solvable  1 

No  defiri'iion  obtained  from  office  1 1 

Low  (green) 

Proven  technology  and  no  problems  S 

So'vabie  4 

Test  p'an  dcvsed  and  tests  completed  1 

Solvab/c  with  no  niaior  schedule  change  1 

No  dt-l.n.tion  ottlained  f-nm  oMcc  10 


HiHli  Kisk  111  v»in<' riff  u  t-s.  ntirnilivc  tcriiis  fitr  liij’li  risk  s|rK>i  ifipr)  a  liiobliTri  as 

solviiltU-  if  file  sclu  (lull'  toiiltl  l)(‘  strolih*  d  or  |HTforin;tin.p  rt'Cjiiirbmeiits 
could  Ik-  m;idc  less  .strinj'ciit.  In  otlitTol  n  -s.  Iiixli-risk  clcnicitts  were 
'•on.-iidcrcd  iiroltahly,  l.iit  not  nciTSsarily,  iins>.;'  tdilf.  In  still  otlicrs. 

Iiij.li  risk  clcrnciKs  |Hiscd  "inajor  itroltlcnis,"  Ittil  flic  source  of  diffi- 
c’llty— cost,  sclicdiilc.  or  jh'i  forintinct' — wits  not  cited  or  tied  to  solv- 
;il)ili(y.  One  office  considered  elements  lii^jli  in  risk  if  tlie\’  ucre 
ctirrentl\  within  the  slate  ol  the  art,  hot  1  other  olfu  es  saiil  hi)'lt  risk 
elt  ioents  were  heyontl  the  state  of  the  art.  Finally,  1  tdfice  descriherl 
risk  as  hitth  if  a  jilan  for  lest  mu  or  manaum;'  (lev  t-loiiment  had  not  vet 
iH-en  devised.  Karly  in  the  ac(joisition  cycle,  test  [ilans  m;ty  have  been 
dev  i v'd  for  elements  that  were  iieii  her  within  the  slate  of  the  ail  nor 
einiicly  new  Testiiin  :s  a  crilerion  distinci  from  the  criteria  reiMiried  in 
other  oITTi  es 
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MiKlcrati'  Ki'.k 


TIr-  tn'ins  tor  dofinin)’  iniMU-rati-  risk  wi'rc  also  incoasistoni.  Many 
oMrvs  iisrd  this  rating  to  nu-an  (hat  iini  crtainty  remained  despite  sonu- 
sneet'ss  in  itevi-lopinent.  Hut  1  ot'liee  applied  the  ratinn  to  elements  still 
iM'yond  the  state  ol'  the  art .  <  )ne  ofl'iee  considered  prohlems  to  ha\ c  heim 
miMlerate  in  risk  it  they  were  solvable,  without  stipulatinji  anything 
alxiut  schedule  tlelays.  Two  other  ort.ees  added  that  sc  hedules  i  ould  not 
slip.  'I'wo  offices  usc-d  tests  as  the  hmsis  fer  ratin>l  risk. 


Ijuw  Ihsk 


There  w:ls  morc>  consist c'ney  in  the  definilioP"  of  low  risk  Many  offices 
c  ited  prcivc'ti  fc-c  finc»lof»y.  no  prohlems,  or  nos/x  c  iaJ  teelinolo>,'y  rc'quired. 
Hut  h  offices  said  prohlcnnatic  clc'inc'nls  could  still  l>e  low  in  risk,  pro¬ 
vided  ihcMc-  was  no  thrcsit.  or  no  major  threat,  to  schedtile.  And  1  office 
usc-d  tc'sts  as  the*  basis  for  qualitative  risk  ratiriffs. 

In  sc'veial  case’s,  detinitions  were*  inconsistent  across  the'  raliiift  c  .itc’t'o- 
ric’s.  For  example,  “bewond  the  state  of  the  art"  was  usc>d  to  dc'scrilH’ 
mcKlerate  risk  in  1  office*  but  hinh  risk  in  4  others.  "Solvable*"  w;ls  the 
definition  of  modc*ratc*  risk  niven  by  I  office  hut  4  others  defined  low 
risk  JJi  this  way.  .MorcHiver.  staff  mc*mlx'rs  in  H  pro^irtim  offices  providc*d 
definitions  that  were  inconsistent  within  catenoric’s.  In  1  office,  a 
rc*si)ondeni  said  hinh-risk  tec  hnoloky  "iis  unproven,  while  another  said 
high-risk  technology  may  be*  within  the  current  state*  of  the  art  (alieady 
[iidven  111  at  lea.st  other  applications )  I.n  another  offici*.  one  resiMindi'iit 
hasc*d  a  clc*finition  of  moderate  risk  on  some  devc'lopmenl  success. 

Allot hc*r  stipulated  th.at  moderate  risk  meant  "beyond  the  state*  of  the* 
ait" — that  is.  the  te*ciinology  had  not  yet  bc*e*n  develoiH*d.  In  a  third 
office*.  rc*siH»nde*nts deliiH’d  low  risk  in  contradictory  terms,  one  citing  no 
problems  and  another  allowing  problems  so  long  as  the*y  wc*rc*  solvable*. 

Thc*se*  inconsist encic’s  within  and  across  rating  categories  imiily  that 
high,  mc’diiim.  and  low  are  not  ad(*quat(*  dc*scriptions  of  risk.  V(*t.  as  ue 
clc*scrilH*d  in  (•hapte*r  H.  many  offices  usc*d  thi*se  ratings  in  their  iirogram 
elocumc*nts.  including  those  on  whii  h  mik'stoiu*  revie*w  dc*i  isions  \\i*re* 
ba.se*d. 

In  summary,  we  found  wide.spr(*ad  ineonsist(*ncy  in  tin*  narrative  terms 
use*el  loele*fine*  qualitative  ratings,  .•\e  ross  office-s.  difle*n*nt  e  i  iti*ria 
(lc*finc*  high.  m<>d(*rate*.  and  low.  .Scime*  re*siKind(*n(s  \M(hin  and  across  the* 
office’s  com radicte’d  oiu*  anothi*r,  as  \\  lu*n  one*  ele*timtion  of  meKleiate* 
coiinicted  with  allot he*r  or  one*  ile*fmit ion  e.f  "moderate*"  \sas  the*  same  as 
a  definitie.n  of  "high." 
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Probabilistic  Terms  for 
Defining  Qualitative 
Ratings 


Progrant  Management  Staff 
Views  on  the  V^alue  of 
Qualitative  auvl 
Quantitative  Risk  Efforts 


Most  of  Itto  in;m;i>ti’mciU  ofl'iccs  used  n:in';iti\  f  tiTiiis  to  dctmc 

ttu'ir  giialit;itiv«>  ratir.ns.  l>iit  usfil  tjuant native  ti'i  rnM — iliat  is.  llies 
tJ.s<'d  a  pnibabilistie  «-stiniate  to  express  tlie  likeliliiKid  of  not  meet  in;; 
IHTl'orinanee  n'<|iiireim‘nts.  \V«‘  askc'd  resixmdenis  wlio  used  ipiantita- 
ti\«*  terms  to  sjKs  ify  ilie  ran;;e  ot  prolialnlit ii‘s  tliey  used  to  H'|iresent 
bi;;ti.  miKlerate.  ami  low.  'I  beir  answers  wen*  seattered  ac  ross  the  ranue 
of  probability  from  zero  to  lull  |H-reent.  The  lower  boundary  lor  ln;;h 
risk  ran;t»‘d  from  In  toSn  iK'nt'nI.  'I'liat  is.  aeeordin;;  to  a  resjxtndent  in 
I  offiee.  a  ebaiue  of  111  ]H‘reent  or  more  lliat  six’eifieatioiis  would  not 
be  met  was  eonsidensl  lii;;ti  risk.  In  another  offiee.  risk  was  not  hi^li 
unless  tli<‘  probability  was  at  least  Kll  iH-reent.  For  nuHlerate  risk,  the 
pri'liability  ranged  from  as  low  as :]  (H'reent  to  as  hi;;h  as  7!l  iH-reeiil . 

For  low  risk,  the  iipjx*r  Iwnindary  varied  from  2  to  Uil  ix>rienl.  Finally, 
ri'sjxmdents  w  ithin  :t  of  these  5  offiei's  cited  ineonsisieiit  qiiaiititalive 
terms  for  risk. 

In  summary,  ineoiisisteiuy  was  widespretid  within  and  across  offices 
thiit  hiist'd  qualitative  nilin^s  on  prohabilistii  estimates  of  risk.  Since  no 
offices  reiHirtiil  having  usihI  quaiititiitivi*  terms  in  the  ri‘\  iew  process  or 
in  program  dm-umentatioii.  reviewers  nitiy  have  wen  only  the  qualita¬ 
tive  ratings  and  may  never  have  discovered  or  resolved  the  underlying 
diserepaiu  ies. 

The  only  clear  diffi-renee  in  the  procedures  tin*  services  used  iiir  rating 
risk  concerns  prohabtlistie  terms.  Compai  ing  the  ■*>  offices  using  Ihesi' 
terms,  2  .Xrmy  :ind  2  .\ir  Foriv  offices,  we  found  that  the  ,\n  Force  set 
more  stringem  .Kmndaries  for  high  and  mocU'iate  risk  than  the  ,\rniy. 
For  instaiKf,  a  <  '••K-m-nt  cininceof  not  mei-ting  iH'i  formam  <■  reqiiin*- 
ments  would  In*  rated  high  in  risk  in  Ixith  of  the  Air  Force  offii os  hut 
iniKlerale  in  2  of  the  d  .Army  offices. 


In  addition  to  obtaining  the  data  on  hovs  ih'*  program  managi'ineiil 
offices  (lefimsl  and  rateil  risk,  we  asked  staff,  w  hen  it  was  apiiropriale, 
for  their  views  on  the  value  of  qualitative  and  iiuanlilali\ c  risk  efforts 
We  expecUsi  ih«'  |>relerenc<,s  of  staff  memlK'rs  to  relied  the  lyiH*  of  risk 
efforts  [lerformed  in  their  offices,  and  this  was  indeed  w  hat  wc  found. 

In  this  surxoy.  few  res|x»mk’nls  ( If!  iH'rceiit  id'  the  'i.)  w  ho  were  asked 
this  quest  ion  I  preferred  quant  it  at  ive  ratings  of  risk,  and  not  many  ( 24 
IK'rei'iit )  thought  the  offices  should  Ik*  required  to  perform  quant  ilalixi* 
assessments  In  line  w  ith  technical  risk  ef foils  in  t  heir  oi  l  ices,  mm  e 
than  hall  (!m-  res(Htinlents  (titl  fH-rceni  I  said  ltte>-  prelerreil  eitlier  a  (pial- 
itativeors  nne  other  h-ss  strucliiri-d  proi'edure. 
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Table  4.3:  Advantages  and 
Disadvantages  of  Quantitative  Risk 
Efforts  Ciiwd  by  ImC  rrograru 
Management  Offices 


The  roiLstins  for  llteso  rating  (.rcfiTiiitfs  warrant  tonsidoralioii,  hfcaii.so 
Ihvy  rovoal  chara'  lfrist jrs  of  llu'  various  approaclios  that  wcrr  ikt- 
ceivod  to  Ik*  im|iortaiU.  Tlu>  n*a.sons  also  tliat  furtlu'r  trainmn  in 

or  supjiort  ftir  tcflinical  risk  assessment  miHht  be  helpful.  Ib'siHiu.lenls 
in  set  eral  offufs  noted  that  (|iiantitative  ratin;'.s  seem  more  hflortnis. 
addinjj  diseipiim*  to  Ihe  asst'ssment  jiroeess  or  heljiinit  define  program 
strueture.  Some  resiKindents  suyytested  that  the  rt'sulls  of  quanlitati\ c 
efforts  are  more  reliable,  meaning  that  they  mote  aeeurately  identify 
risks.  Hut  many  others  said  that  it  is  difficult  to  express  risk  in  quantita¬ 
tive  terms  or  to  ajtply  one  quantitative  mcKiel  across  several  different 
programs.  Many  said  that  quantitative  efforts  ri*quire  resources  (staff 
or  tune)  not  .dways  readily  available.  Ami  some  claimed  that  the  results 
of  quantitative  efbirts  are  ruit  n-liable  heiausi'  tlu'.v  cunnol  lx-  dcfiemled 
on  to  identify  risks.  Overall,  the  resiiondent.s  in  the  ‘in  offices  were  twice 
a.s  likely  to  cite  the  disadvantagi's  of  quantitative  risk  efforts  as  to  i  ite 
any  advantages.  (.Sts>  table  4.1}.) 


Number  of 
Offices 

9 

3 

3 

t 

1 

1 

1 

12 

9 

6 

5 

5 

2 


Opinicn 

Advantages 

Add  riQor 

Help  delino  piogram  struclutc 
A^e  rehjiyie 

Help  111  estimating  p'Cg'ani  costs 

Contorm  to  Slanda'd  engincennq  dppioach  In  nsK 

Help  Suppoil  piogiam  dccsions 

AMoa  tlexibilily  m  ralmg  fi$k 

Disadvantages 

Bequiie  lesou'ces  not  ai^a/s  ava  iauie 
Use  leims  ha'd  to  dei'nc 

Require  applical.on  ol  Ibe  same  modal  to  dillcrenl  programs 
Arc  not  reliable 

Reduce  decisionmai>eis  (ieiit;i!ii/ 

Do  not  produce  t'mci/  results 
Lead  lo  m/crominapemri  t 


When  wi-  aski-d  the  program  management  staff  aUitit  qtialitative  rat¬ 
ings.  the  iirimary  tidvanlage  they  cited  was  the  reliability  of  the  results 
Kespundeiits  in  ’J  offices  noted  also  that  the  results  from  (pialitative 
effiirts  are  more  innely  than  those  Ironi  (|uaninali\e  efbnts.  Tlie  pri¬ 
mary  disadv .image,  ar-eordmg  l:>  others,  is  that  qualitative  rc'iilis  are 
Hot  reliahli* — they  are  too  suti|ei  t  i\e  or  imprei  ise.  I-a en  vi.  ar  loss  i  he  J'l 
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Table  4.4:  AdvantaQea  and 
Disadvantagea  of  Qualitative  Risk 
Efforts  Cited  by  the  Program 
Management  Offices 


offift's.  rf.s|H)n(Ji‘nis  wi-n*  twice  ;ls  likely  to  cite  advaiUaj’es  of  oualita- 
tive  elfort.s.  reversmH  the  pattern  for  c|iiantitative  effons.  (See  table 
•1.4.) 


Opinion 

Advantages 

Are  reliable 

Produce  timely  results 

Correspond  to  convenitonal  risk  concepts 

Are  comprehensive 

Produce  results  that  are  easy  to  communicate 
Use  resources  that  are  available 
Are  acceptable  to  stall 

Oisadventages 

Are  not  reliable 

Require  resources  not  aiwa/s  available 
Do  not  prrxi'jce  I  rntly  rc,S-"S 
Are  not  comprehensive 


Number  ol 
offices 


6 

2 

I 

1 

1 

1 

1 

4 

t 

1 

1 


These  differences  in  perceptions  are  not  in  thentst'U  es  problematic,  hut 
they  do  stitytest  that  the  avtiilahility  of  variotis  a|>proachts  to  technical 
risk  assessment  (in  th<“  handtxwtk  of  the  liefense  Systems  Management 
(’oll<-;:c  on  risk  assessment,  fort'xample)  is  not  enough  to  ensure  that 
nrogrtim  offices  will  adopt  any  partmilar  approat  h  or  rely  on  the 
results. 


Sunmiarv  of  IX’fillitions  f  ew  of  the  (nogram  m;inag<iment  offices  kntnv  how  ix>i)  or  .sen  iee  docu¬ 

ments  deline  technical  risk,  and  h‘W  Itad  Ihcii  oy\n  [xthey  formally 
defining  tet  hnical  risk.  .Many  offici's  nonetheless  had  a  definition  shared 
informtilly  by  till  or  most  staff  members  within  an  offit  e.  but  the  defini¬ 
tions  \  aried  widely  from  office  to  offic  e.  Some  were  i>redic;itc'd  on  the 
likehhoorl  that  technical  problems  would  arist',  ti' hers  on  the  hkelthtxid 
that  inriblr'ins  woold  jiriv  :mtl  liuid  to  program  fiitlure.  .Some  considered 
Ih"  likclihtrrxl  r)f  fiiiliire  within  cost  or  schediili'  constraints;  otb.crs  did 
not.  Finally,  in  si\  mstam  c-s.  there  were  no  (  (insistent  dcftnil tons  of 
technical  risk  even  wiihm  an  olfice.  .Since  few  of  the  offices  were  atcare 
of  an>  Ik  Mior  service  del  mil  ion  of  leclmii  al  risk,  and  si  me  I  he  dt'fmi- 
lions  that  (III  exist  ate  mnlngiidiis.  the  inconsistency  we  found  in 
wiirkiiiK  (lefimtions  is  nut  surprising. 
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Many  nlTiccs  rxim-sscil  i  isk  in  (|ii.iliiaii\(’  ratinji- — hijjli.  iiiii.lcratf.  .ind 
low  (ir  ri'd.  yclliivs .  and  UitM-n—  and  ratin>is  were  dflincd  in  naria- 

Ii\  (“  or  ^iiiant  it  at  i\«-  torms  Kor  «‘xain|iln,  liijih  risk  was  somciniu's 
dcliiu'd  nartaln«‘ly.  as  in  "iM-yond  llio  stain  of  thn  art.  ”  or  ((iiaiiiila 
ti\  (‘l>',  as  in  "ill  Inasi  an  So  iM-rcnni  i  liaiu  o  o*' failure."  U'll Inn  S  of  I'lees. 
qinmlitiiliviMlcliniiions  were  iiK  onsisteni :  oiir  resiMtndents  s.-t  dif  ferent 
Imtindaries  for  file  s;mie  lexels  of  risk.  Hut  n;irr;ili\  e  ;i.s  well  .is  (jiiant iiii- 
live  definitions  were  widely  d.na-rjjent  aeross  iill  of  fiees  :ind  were  often 
eont  radii  lory. 

Willi  definitions  and  ramies  so  ineonsistenl,  eonfiision  is  almost  ine\  i- 
lidile.  I'or  e\  imiile.  one  staff  nienitier  m;iy  sity  iliat  risk  is  liiiih  lH-eaii.se 
the  teehnolo;fy  is  tinproveii  Another  niiiy  siiy  risk  is  low  iM-eaiise. 
althon^fh  thi*  lechnolojfy  is  ll•loroven,  no  .vriotis  prohlem  is  exiK-eled  as 
loiif*  as  time  iind  fundiiif'  are  avail;ihl<>.  Still  iinother  staff  memlH-r  mi;Jlit 
see  risk  as  moileniie  hei  iiusi*  no  serious  problem  is  exiK-cled  hut  would 
also  siiy  thill  fiiilure  of  iiny  element  would  stop  program  pro^fress.  More¬ 
over.  where  (iiiiinfilative  terms  iire  umhI.  a  flO-ix-n  t-ni  eluin.  e  of  not 
nuH'tinn  s|H‘<  ifieiilions  is  l  iilleil  low.  miKlerate,  or  hi^th  risk,  deiK-ndinU 
on  whii  h  offiee  makes  the  raiinjl. 

The  results  of  a  risk  effort  in-rformed  w  ithout  rejiiird  forsiieh  inconsis- 
lencii's  w  ill  not  he  very  valuiihle  and  nuiy  mislciid  dei  isionmakers.  For 
'.'xami'le.  i)iu>;iam  staff  may  In-lieve  tluit  a  IKl-jx-ri  ent  risk  is  low.  while 
diH  isiommikers  see  a  fltl-iH-reent  risk  as  hijfh.  If  review  diK  timenls 
sim|)l\'  reiHirl  low  risk  (Horn  of  those  we  exiimined  h;id  ijiianlitiitu  e  r;it- 
in>ls).  dei  isionnuikers  imiy  never  know  ihiit  the  eslinuite  of  risk  Wiis 
aeluiilly  an  estimate  of  dll  iK-reent.  1-ven  if  inconsistencies  iire  hiter 
uruinered  ;ind  resolved,  lime  will  hiive  Ix-en  lost.  Fiiilheiinore.  ;dthoU)’h 
resiMindents  w  iihin  nuiny  offices  did  use  consistent  definitions  of  risk 
and  risk  ratings,  the  inconsistency  across  them  imikes  it  v<‘i-y  difficult 
for  re\  iewers  outside  iiny  office  to  eviihnite  the  results  of  risk  efforts  or 
to  coinpiire  results  across  pronnims. 


The  Communication  of 
Information  to 
Decisionmakers 


In  I  he  follow  inn  discussion,  we  apinoach  the  iiroblem  of  communical  tnH 
technii  al  risk  information  from  the  .separate  (K-rsiH-clives  of  the  offices 
and  t  he  reviewers.  We  examine  I  he  s|)ecific  i-vsues  of  access  to  informa- 
liun  and  its  ade<iuacy,  content,  and  o\er:dl  pri>senlation. 
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All  olTiccs  rfiMulcfl  iisinn  ilu-ir  'isk  criorls  in  (ir<i^;raMi  inanajicmcnt  In 
2a  (il'IV cs  (!t2  (xTiiail ).  \v«-  uoff*  1»)1<1  ilial  i  isU  •■Hurls  had  Ims'H  iisfd  to 
siipiMirl  t(‘«  linu  al  decisions  siu  h  as  seleciin^;  design  alteriiai  i\'es.  pro- 
Hrain  si  hedulinn  and  n'strncliirinn.  and  assininnn  tasks  to  groups 
outside  the  projtram  otfiee.  S»‘Ven  ol'fiees  ( 2S  |HTeent )  also  cited  t!a'  iisi‘ 
of  risk  efforts  to  stippoii  dei  isions  on  overall  pni>:ratn  cost  or  a|)jihca- 
tions  for  fittids  to  e«iver  problems  ident  iTied  hy  tlieir  risk  •■Morts.  finally, 
1  office  re|Ktrled  iisin)J  its  risk  efbiH  lit  the  evaluation  of  vendors'  pro- 
IKisiils.  ha\  inn  applied  it  also  to  tei  linu-;il  decisions,  'flu-  staff  memlters 
we  interviewed  —  pronrain  iiiaiianer.  deputy  ^)r^^nram  mananer.  i  hief 
foniiuvr.  ami  V  out  fact  officer — played  ke>  roles  in  the  daily  ofx-ralions 
of  a  pronram  If  teduiical  risk  informaliiai  is  to  Iw  used  in  the  pronram 
<J(s  is'oii.s  di'serilK'd  ah<»v«-.  ihesi-  indiv  idiials  must  lx-  anare  of  and  ha ve 
access  to  this  information.  In  addition.  the>  must  have  enoiinh  knowl- 
edne  alxMit  risk  efforts  to  understand  the  results  and  their  limitations. 

The  imjxa  tance  of  the  pronram  mananiT  in  both  the  pronram  office  and 
hifilier  review  jinMesses  makes  this  individual  s  knowledne  alxuit  tech¬ 
nical  risks  id'  particular  eom  erit  the  projiram  office  level,  pronrain 
maiianers  have  |>rimary  res|)onsihility  for  daily  decisions  and  are  in  a 
IHisition  to  reciuest  a  risk  effort  and  ensiiie  that  ihi'ir  tei  hnical  staff 
know  atxHit  It.  SMieii  preparmn  pronrain  dcH  iiments  and  when  hriefmn 
(hvisioninakers  at  pronrain  reviews,  a  pronrain  mananer  must  address 
I  ho  »iuestion  of  risk. 

Thus,  in  our  interviews  with  iironram  maiianers.  we  asked  not  only 
wlu’ther  they  were  aware  of  the  risk  efforts  |H'rfonned  for  their  pro- 
nrams  hut  also  whetiu’r  or  not  ih<‘  pronram  mananers  knew  how  an 
effoil  had  been  iK'rforr.n*d.  Sjx*cifically.  we  aski-d  the  pmnram  m.ina- 
ners  whether  they  knew 

•  forinat:  Were  risks  rati-d  in  qualitative,  ()uantitalive,  or  narrative  terms'.* 

•  focus  on  the  system  as  a  whole  or  on  subsystems'.’ 

•  ii'put  obtaiiM'd  from  one  individual  or  a  ni'o  ip'.’ 

•  sources  of  data:  Uid  havinn  le<  hniial  risk  informaiion  dejH’nd  on  the 
cont  raclor.  lalioralory.  pronram  tdfice.  or  ol  her  sources" 

•  approach:  \\'<-re  quantitative  or  qualitatn  e  approaches  (such  as  those 
•  lesci  itieil  in  •  hapter  2 )  iisixi  to  determine  risk" 

Ilaviiin  lliis  knowhslne  would  help  pronram  mananers  understand  and 
e\  aliiale  t  he  results  of  1  he  risk  ••fforl  and  enable  t  hem  to  make  well- 
informed  reports  to  re\  lev  el's. 
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\\V  fdiittd  that  tiu'  prdfjrant  iii;uia>!(Ts  vvcji‘  avvan-  of  the  jn  iniai y  risk 
♦'fforts  for  thoir  i»‘(>>;raji)s,  and  most  knew  tlio  forniat,  mojk',  piiM  oilm  r. 
sources  of  (iata,  and  approach.  Ihit  of  tlic  lio  iiiaiiajlcrs  \vc  sur\  i‘\  i'<l, 
sonic  did  not  have  eoinplcti*  information  For  exampU*,  ">  mana;;crs  did 
not  know  the  approacli  that  had  lici-n  list'd  to  asst'ss  risk,  and  ol  these.  1 
also  did  not  know  the  format  for  retxirt inn  risk.  I  did  not  knowitiepio- 
ei'tlure  for  eolleetmn  tv'l'ideal  risk  information,  and  2  did  noi  know  the 
sonrei's  of  data.  Two  other  m.inakers  knew  the  approai  h  hot  did  not 
know  other  iLsfKs  tsof  how  risk  had  Ixi-n  a.sse.ssed;  1  did  not  know  tlie 
si-ojM'  atid  1  knew  neither  the  format  nor  the  pro<  ediire  .Sm  h  informa¬ 
tion  is  often  im|xirtant  in  manannin  and  appraisink  the  status  ot  hinh 
feehnolon.v  systems.  IVonrani  managers  w  ho  lack  this  inform.iiion  are 
tlii'refore  aide  neither  to  fully  ♦•valui'te  the  results  of  their  risk  assess¬ 
ments  nor  to  (lest tiIh*  th.'ir  assi'ssments  fully  and  prom(itly  in  the 
review  pna  ess. 

Althounh  our  analysis  of  interview  data  from  otht'r  its  hnii  a!  staff  mem- 
liers  was  not  as  detailed  as  our  analysis  of  data  from  th<-  managers,  the 
results  reveal  that  some  individuals  had  liltlt'  or  no  informal  ion  alMiut 
risk  efforts  in  the  (irofjram  manajjement  offices.  In  !•  of  oiii  Jo  tdlices. 
there  was  at  a'ast  one  )K‘rson  who  did  not  laenlior.  a  lisl;  efforl  '.hat  was 
deserilHsl  hy  others  in  tlie  office.  Furthermore,  some  of  the  flaps  and 
inconsisieiu  ies  m  our  data  indii  atea  lack  of  eommiinn  .iiion  alioni  the 
risk  efforts  For  example,  at  lea.st  one  staff  niemtier  in  eai  h  of  1  td  fices 
did  not  know  the  lormal;  in  each  td'  1  offices,  at  least  one  did  not  know 
the  scoiH';  and  in  each  <d  o  (d'fices  at  least  one  did  not  know  ila-priHc- 
dure  for  the  (irimary  risk  effort  on  the  system  thi  office  was  n  s;Hinsilde 
for. 

Our  resjHindents  (-ouid  pot  Im*  exjxs  ted  to  know  all  the  detars  <if  the  1 1  k 
effoi’ts.  However,  the  individuals  we  i.ilerv  icwid  (anioiif'  them  (ii'|>iii\ 
pronram  manaf;ei>  and  i  hief  eiiftmisTs)  w  ho  are  in  chaific  of.  or  f|i\e 
input  to,  a.s(H'(  Is  of  the  develojaiient  o'  systems  should  know  al  least 
what  efforts  have  Ims'ii  performed  and  have  access  to  relal'vely  detailed 
intorinatioii  about  them  Otherwise,  it  v.  ill  lx-  diffn  iill  to  mamlain  <  iear 
jiriorilies  for  the  lei  hnical  asj>ecls  of  system  developiiu'iit 

Hecaiise  of  I  he  effe<  t  that  te<hni<al  risk  can  have  on  i  mit  r,i<  i  (N-ciskhis, 
we  inlerv  iewed  the  proflrams'  contract  oi  l  a  ers  in  order  to  deieriiune 
how  liiev  learn  alxnil  technical  risk.  Two  sanl  they  flot  no  tci  hnic.il  risk 
iiitormat  ion.  ami  the  ot  heis  said  they  learned  alxaii  ici  hnii  a  I  risk  ni 
brief  lints,  iiroftraiii  documenls.  nr  ininrm.il  disc  ussmiis  with  the  pi  c  iftr.im 
(d  I  Ice.  1  low  ever,  as  we  ;'e|)oi  led  m  chapter  -'t.  not  all  proftr.im  doc  oiiicni  s 
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and  briofin^;s  inchulfd  technical  risk  mhnniation,  and  when  they  did, 
much  reniaincfl  uncertain,  su<  h  as  the  sct)[M‘  of  the  ratinjjs  and  what 
they  mean.  Inforitu  I  diMUssions  and  meetinjts  may  not  In*  more  t om- 
plete  than  tliis. 

Wc  found  differenc«*s  lK‘tW(H‘n  the  s<*n.Kvs  in  staff  knowledge  alxiui  risk 
efforts,  a  ftreater  projKirtion  of  Army  staff  lai  kin)'  knowledge  than  staff 
in  the  two  other  s«*r%'ic«‘s.  More  pronran,  offices  in  the  Army  had  staff 
members  who  did  not  «  ite  a  risk  effort  dtscrilasi  by  oliicrs,  ev  en  when  it 
had  iHs-n  diKiimented.  And  more  offices  >n  the  Army  had  staff  niemlH-rs 
who  did  not  know  the  formal,  sco|h-,  or  prcKtshirc  of  iheir  risk  effons. 


iX'cisionmakers  at  hifjher  review  levels  obtain  infonnation  on  teclinical 
risk  from  mik*stone  briefiii)is  and  diH  iiinents  provided  by  the  pro>;ram 
office's  (Inly  about  b.alf  the  brH‘fm>>  charts  wt“  obtained  from  the  pro¬ 
gram  offices  in  our  study  even  made  reference  to  techniial  risk,  Al!  that 
did  uM'd  ratings  to  do  so,  onlv  one  providing  information  on  scojh*  and 
anotlier  on  approach.  To  obtain  additional  information  alaiiit  tis  hnical 
risk,  a  reviewer  would  have  had  to  a.sk  the  program  manager  for  it  sjk*- 
eifically.  Vet,  as  we  indicated  earlier,  program  managers  might  not  have 
been  able  to  go  into  further  detail  atxMit  results. 

In  addition,  we  found  two  prohleiris  witli  reviewers’  reliance  on  program 
diKuments  for  tis  hnica!  risk  information  Fii'st,  as  ilescrilK-d  in  t  h.i|itcr 
3,  many  dm  iiments  (ontained  no  ilisciission  of  ri.sk  or  an  incompicte  one. 
For  example,  some  program  dm  uments  incliidisl  an  ov  erall  rating  of  a 
system's  ti'chnical  risk  but  no  explanation  as  to  what  tin-  srojK*  of  the 
rating  was — that  is.  what  part  or  i-arts  of  the  system  had  lH>en  consid- 
ere<l.  The  lf»8.'>  dcs  ision  ixKirdinaling  |)a|x  r  uixlatc-  for  the  Air  Force 
N.WSTAK  t  M  T  t>niii.ii>.-ni  program  contained  an  overall  rating  of  risk  for 
the  system  without  an  explanation  of  winch  subsystems,  if  any.  1.  ul 
bivn  eonsidereil  in  this  determination  Snne  program  offm  s  used  a 
qualitative  format  for  risk  ratings  in  their  diK  imients  hut  provided  no 
narrative  of  what  they  meant — an  I’xample  is  the  Nav  v  s  \  -'JJ  (Kprey 
acquisition  (ilan.  A  thinl  example  of  incomplete  discussion  of  risk  is  the 
listing  of  risk  Hems  only  with  no  explanation  of  how  or  vv  hv  the  it<  ins 
Were  <  hosen.  Tei  hmcal  risk  was  presiailed  in  this  form  in  the  acquisition 
plan  for  the  Nav  >  s  Submarine  Ailvanceil  Comhat  S\  stem 

SiMoiid.  the  program  d<«unienls  vve  reviewed  did  not  pri‘s<  nl  technic  al 
risk  inrormation  in  any  standard  way  The>  vai  ioiislv  presented  risk  in 
((iianlitative.  (jiialitatiM'.  and  narr.itive  terms.  iis*-d  any  sc , tie.  and  gave 
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.I''  luiu  It  ( ir  its  liti  1(‘  til  •tail  as  i  lif  |ii(%;i  aiit  i  il  I  iti-s  i  I  .i  tsc  I  i  \  f  |  n  i>i;t  .tuts  tit 
mil  sailt|ilf  list'll  till  It  fi'lH  liiMI'ills  liinil  lliM  lllliriil  ti' 'liK  lllltflll  F'nr 
I'vanH'li'.  iiii  I'aiiy  mtsiiih  uI  iln-  ;ii niiisii imt  plan  Im  tin-  Air 
Slinrt-K;tn>;i'  Alt;ii  k  Missili>  n  (mrti'il  risk  in  ijiialiiai  Ar  i  nis,  luii  ;i(i 
ii|Kl;iti‘  III  till'  siiini-  il<M  iiini'iit  usi'il  ili  sri  ipi  i\  r  iiTins  mtly  II  is  iiiii  li'ar 
wlirthrr  lit!'  risks  mi  rt'iisitl.  ilnniinstn'ii.  nr  siasril  llir  sariu' 


umniary  of 
’.ommunication 


In  I  iirii^riiins.  iltf  niiiiiiM-r  nl'  rati'jinni's  iisi'il  in  l  iUi'  lit  liini  iil  risk 
(  I'ankml  In  niti' (Inn’iiii'iit.  |nr  f\ani|>li'.  ita  \  JJ  <  ispri'S  (irni^rani  nsisi 
ratings  nl'  Inw.  low-ini'iiniin.  iiu-ilniin.  'iircliiim  fiikii.  hikI  Invili  In  rrpri'- 
si'iii  risks  !n  iinnilii'r  ilm  iiini'nl  I’nr  ilu'  sii;ia'  pm^rafn,  Itii'  raiinjis  wi'if 
low.  niiKlt'raU'.  ini-iliiini.  ami  hijili  “MiMli'raf<'"  anil  ■■nii'ilniin."  iwn  ilif- 
li'ii'ni  |K  lilts  on  tins  SI  all',  ui-rr  tri'ati  d  syn')n\’niniisly  ■.'Isfwlii'ri'  \S'liat 
was  iiu'anl  li>'  I'at  li  sralf  was  not  ili-si  i  iIh'iI.  nnr  vvas  il  i  lnar  Itnw  In  rmn 
part'  till' t wn  St  iili's. 

Ftin  iti'r.  for  n  of  iln*  'J't  prn;irains.  r;ii  mils  for  oin'  snlisysifni  t  lianni'il 
vs  itliniit  I'XpiaHiitinn  In  llir  A<l\ iim  t'd  lli  ln  npinr  linprovi'nii  iii  l‘ro- 
^ram,  for  I'.vanipli*.  thr  Iransnnssinn  was  r  iti'il  ;is  it  innilrriili'  risk  in  ilit' 
It't  Imi'.'iil  risk  asst'ssini'iii  n'|>nrt  l>ni  as  ;i  inw  risk  in  I  In'  I'lii'rirati'tl  pro- 
jtnini  suinniaiN  anil  in  tln'  ..:tiins;pnn  iilan.  'riti-st'  ilm  nini'iils  wnm  pri- 
piirril  for  lli<*  siinu*  nnli'snim'  ri'vu'w  .  For  ni  illii  i  .\iiii’  nor  any  of  tin'  1 
ollii'r  systi'ins  Was  tlii-r«'  a  iliHuntrnlrd  r\pl;tnaiinn  tor  i  tiiin^tinn  llti'  liit- 
in>ls.  'riins.  111!'  task  of  n't ouni/.init  llu'  i  lianvir  and  ri'(|in'siinn  iiildiiinnai 
inlormitlinn  liad  Ins'ii  k'li  m  tin'  rt'\  ii'wms 

('liaiik' s  in  Ihi'  Wiiy  risks  wt'it*  pri'si  nii  il  in  ilir  ilm  tinn  nls  may  lia\i' 
rt'stilli'il  (rnltl  a  ri'liu  laiut'  In  lili'itllly  St'i  imis  prnltli  lils  SnltU'  stiiff 
.ni'inlM'is  said  Ihal  ratfis  nlti'ii  Itrsit.iii'  In  rr(>nn  '  risr’  nr  "Itiiili  risk  to 
ri'vii  wt'rs.  prt'li'rrinit  Inwi-r  ralinus  r\  m  vn  In'ii  i lti'>  an-  not  ajiproprialt'. 
('lianj;in>‘  litt'  risk  tiii'ivjs  in  pmurain  ,1m  iinti'iilatinn  and  i-Mfiidin^t  lli" 
riit  iit^  St  iilrs  may  Im'  w;p\  s  nl  it\  nnlini;  Initli  i  isk  arras 


Wr  .'ntind  that  Ilit'  iipplnarll  nl  Itini’.ralil  manarti'mrnl  nlfii  rs  In 
adtlirssiiiii  irt  liniral  risk  olfi'rrtl  im  ihal  mlnriiialinii  wntild 

Im'  prn\  idisl  In  dri  isinntnakris.  w  il  Inn  I  In  nl  I  n  rs  nr  ai  t  It"  hi>tlii'r  li'\  rls 
of  ri'\iiw  .Mnsi  prnjtraiii  iiiiiiiakri  s  .nid  in  linn  al  s|al!  Inil  nnl  all.  wrri' 
awarr  nl  Itir  I  liaiii;  iirisiii  s  Ilf  III!  rr  risk  i  llniis  iin  Imlmt;  Ihr  Im  mat 
fnr  ri'iKirl  ink  risk,  l  hr  s.  njH-.  i  hr  prm  nhit  r-,  I  nr  I  nllri  link  Its  linn  al 
data,  till'  V  Mill  os  nf  In  hull  a  J  mini  in.il  'on.  and  I  In'  ajipmai  h 


Paiti-  M 


rKMU  H*» ’»  Tr<  hdiral  Ki*k 


ogi'iun  ManageiiKMU 
Tice  Staffing  aiui 
aining 


cftinf*  and  ('ondiu  imU 
ik  Kfforts 


i'ha^tirr  t 

%»tt  ^  (  unriit  Iw**  (*>  (hr 

of  In  hitiml 


Ni'iilirr  tl'c  iku  iIh-  lindino  i''\  I'^ucd  \m  ic 

.i(lc»|uatc  1(1!  iMliiriiiiii^  |•|■(l;^l .(III  si. ill  <<r  ha  kavcis  .iImuiI  Us  Imit  .i|  r  isks 
111  sonic  In  icliniis  ;ni(|  iI<h  uincnl.s.  us  Imu  al  I  ’sk  w.is  no!  iacii  addressed 
111  (it lid's,  risks  were  treated  mnmn.nl.v.  as  w  lien  a  ->  siein  w as  eii  a 
(|U, 1 1  It  at  i\  (•  risk  ralintJ  w  itli  no  exj.l.tn.ii  loii  ( 'oniiilic  il  invl  t  lie  ha  uam-is' 
U'lsk.  diri'ereiil  d(K  uineiits  addressis)  risk  dif  lereiil  )\-.  l  alinji  scales 
etiaiined.  and  ralitijis  chaiijlerl.  all  u  illioiil  ev|i|.ination 

Det  isioiunakiTs  w  al.in  the  |>Hi,i;rani  oftices  and  at  linltier  ha  k  w  hAels 
<  iiiiiinl  base  decisions  on  the  true  tis  hnical  risks  nl  a  system  il  lliey  do 
not  know  al'oul  an  assessment,  nor  i  an  t he.\'  do  so  w  I.en  i liey  are  not 
Hiven  eiioiifth  information  to  evaluate  or  understand  it  ritimaiely.  risk 
ef  forts  that  (fiH  tsionmakers  cannot  use  w  ill  not  1h- ellectiM’. 


In  our  inierA  iiAr  s  w  ith  proijram  offi.-iAi,  we  aske<}  who  w as  itn (»)\  ed  m 
Sl  id  tinn  and  |K'rlormin>;  tla*  risk  eflorls.  We  also  inter\  kamsI  staff  in 
service  schools  to  determine  what  tramini;  was  availahle  to  suii|>oH  the 
I'Hinram  oftne  staff  involvetf  in  risk  efforts.  In  he  ma.iorii  v  of  ...  pro- 
ffram  offices,  we  found  that  staff  wc-re  involvdf  m  Ixith  selci  timJ  and 
jierformitiii  risk  efforts  fnit  that  ser\  ict*  sehixil  traimnu  for  the  assess¬ 
ment  of  Id  him  al  risk  was  minimal.  Thesi-  results  are  iiresentist  in  del.iil 
Ixdov 


In  I'J  proifram  ottnes  i  IS  iiercent ).  resjxinsihilily  for  m-Iim  tiiii;  the  an.i 
I.Mic  al  .ii'i'roaeh  lor  a  risk  effoil  resteil  al  least  jiailly  with  the  t(s  hnical 
siaff  In  li  of  ficc-s  ( 'J-l  lei'ceiit ).  lh(‘  projfram  manager  w;ls  also  iin  oh  ed 
Kes|>ondetus  in  !•  offices  i  d*'.  jxTcent  I  said  that  contrac  tin's.  lalxiratory 
refireseiitat  i\ es.  advisor.-  jianels.  or  oi  lid's  outside  tfie  office  jiartici- 
fialed  III  the  .'x-lcs  tioii, 

( ftice  an  ef fort  li.id  Ixs'ii  vlei  ted.  w  ho  ai  tiiall.v  dnl  the  wi irk  ’  In  IS 
office's  l72  (K'rcenl ).  it  was  tei  fiiin  laiis.  emfiiid'rs,  tir  other  stall  The 
I'Hiiiram  manager  was  iiird  tly  invohdl  in  assi  ssmenls  for  7  olficesCjS 
( cut )  In  d  of  I  !  'es  (  1 2  jiei  i  (lit  I  siiji)mrl  St  al  I .  sin  11  as  I  list  or  sysi  enis 
anal\  sts.  assisted  ui  the  eflorls  and  in  S  i  dj  |x'rcenl ).  prune  i ontrai  tors 
or  siiiijsirt  coni r.u  till's  p.iiin  ip.iidl,  .Sx-m  ii  offices  (2S  jx-rceiit )  used 
iiijiiil  irom  l.ilMir.itory  st.ill  or  ad\  isory  panels 

-\s  for  dll  ferences  ut  the  scr  \  ices,  we  found  I h.ll  .\l  m\  [ili'ilr.ilil  1 1 1.111. 1- 
kd's  weie  rie\er  iiuohed  in  si  |(  (  iimf  or  coiidin  1114'  ri'k  elhets  Siaci  d 
\.i\.\  and  .\ir  Kori c  proki.im  m.inak'-rs  di'l  p.tnn  ijiale.  |i|(i!i.ihl\ 
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condiictiiijf  risk  efforts,  tci  h- 
'ition  ill  the  services'  traiiiinj; 
■illy  defined  as  "[irojtrain  risk," 
.'lie,  niaiiaftenieiit.  or  eoiitrai'tor 
)  risk  e;;eliisi\  (‘ly.  teeluiieal  risk 
•  ■  of  tlu’  seliools  laiiUhl  qiianlHa- 
leal  risk  iLssesstneiit . 
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■  used  i-otirses  iii  otluT  substaiitne 
\ir  Foru’  Insti’  ite  of  Teehnolo(>y, 

'  es  on  reliability  and  nniiniain- 
lool  included  a  discussion  of  risk  in 
and  eontratls  Siinilai  1\'.  llie 
■;fe  I'overed  risk  in  its  projiram  nian- 
^/sties  .Manaklenicnl  ('enter  disenssed 
n  mana>tenient,  cost  anal.ssis,  and  s\s- 
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inar  on  risk  : 


nt  ('entiT  decoted  <  otirses  s(>ei  ifi 
regular  eiirrieuluin  Since  the  earl> 

■  veek  course  called  "dccisiDii  risk  anal- 
ilysts  ;ind  "di-cision  risk  anal\  sis  for 

■  ms  concerned  with  logistics  Tlic 
ic^e  recentfv  offi  icd  a  J-1  2  da\  sene 
am  mana^ters. 
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meal  risk  varied  from  si  bool  to 
.‘11  risk  I’crsonncI  at  I  be  .\ir  Force 
li  risk  IS  discusscil  in  t  lie  t  oiirses  but 
into  sis'cillc  catcitories  ol  risk,  such 
ormaiic-'  TbfNa\  al  I’osi  ^r,idu,ili' 

"  and  talked  about  risk  citlicr  cca,.|- 
1  ISC  Systems  .Manarlcment  ( 'ollc,.'.!.  also 
'  broke  risk  lilt  O  t  osl  ,  si  bed  I  lie,  and  pel 
\ork.  bowiAi-r,  disi  nssioii  ol  risk 
'  Snnilarl\  ,  t  lie  Ai  nu  1  .or;isi  h  s  Mainere 
lie  iiriibabilit  \  ol  iioi  meet  in;;  i  osi 
is  "  but  t  re.iteil  risk  mill  e  itcui 'i  ,ill\  in  I  lie 
iltd  s\ stems  ailal\ SIS  courses  l)ni\  III  I  be 
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CVnttM’s decision  risk  aicalysis  courses  was  the  discussion  of  risk  more 
spc'cific,  aitfiounh  it  dealt  mainly  with  costs  and  schedules. 

None  of  the  service  sclunils  discussed  apfiroaches  for  a.ssi‘ssin).i  technical 
risk:  if  it  was  mentioned  at  all.  it  was  typically  only  descnhc-d.  We  found 
two  exceptions:  at  the  IX-fense  Systems  Manajfement  College,  issues 
associated  with  tc'chnit  al  risk  were  discusscsl  in  one  of  the  management 
course's  and  in  the  risk  managt'ment  seminar,  and  at  the  Army  l>ogistics 
Managemc'Ht  Cc*nter,  technical  risk  was  discussc'd  in  cost  analysis 
course's  as  input  to  the  mVi.  t'slimate. 

When  we  asked  school  staff  memtx'fs  to  rate  their  services'  supixirt  for 
tc'chnical  risk  training,  their  ratings  ’•eflected  the  amount  of  risk  assc'ss- 
ment  training  eaeh  scIkkiI  offered.  Ratings  were  high  for  the  Army, 
wiiich  reporte'dly  gave  a  "gre'at  deal  of  suppiort”  to  the  Center's  efforts 
"lattle  or  no  supixirt"  ratings  were  given  to  the  Air  Force  for  technical 
risk  1  se'ssment  training  at  the  liisiitute  of  Technology  and  the  ,\avy  for 
training  at  the  Postgraduate  Schixil.  Mixierate  joint -service  sujiixirt  was 
said  to  1k‘  given  to  ti'chnical  risk  assi-ssment  training  efforts  at  the 
IK'fense  Systems  Manage'nu'iU  College. 


Hata  from  the  si-rx  ice  schools  suggest  that  technical  risk  assessment  nas 
received  httU-  atti-ntior.  in  the  curriculum.  The  ,\rmy  was  the  only  ser¬ 
vice  that  offered  a  coursi-  on  program  risk  as  part  of  its  regular  course 
offerings.  In  coursi's  in  which  risk  was  mentioned,  and  even  in  >niii  ses 
devoted  to  risk,  technical  risk  was  not  a  fiKiis  and  neither  were 
ajiinoaches  to  technical  risk  ass<'ssment.  The  discussion  co\  ered  either 
s<  he<hile  risk  or  cost  risk  (;r,  more  typicall\’,  program  risk  m  general. 


Prime  contractors  for  the  major  systems  were  responsihli'  for  man\’  of 
the  technical  risk  efforts  descritied  hy  program  offices.  Of  1  he  'l~i  jiro- 
gram  offices  in  our  study,  8  t.'id  [H'rcent )  relied  on  their  prime  contrac¬ 
tors  for  primary  or  other  risk  efforts.  Of  these  8  offices,  ti  had  re(Hiired 
the  effort  Ml  t he  original  projiosals  for  so' m  e  selection  and  1  had 
requiri'd  il  as  a  "ci.ruract  deli\ crahle"  The  reason  for  the  other  con- 
traitor  effort  was  not  siK'cified. 

Of  the  17  programs  ilnit  did  not  rely  on  pi  ime  coni ractors  tor  their  risk 
elforls.  1 2  nonei hetess  used  technical  ;  isk  ii'lormalion  supplied  1>\ 
|irmic  conirai  tors  .is  input  to  their  own  eflorts  For  example,  the  Navy 
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proj^rain  olficf  for  tin-  .Inint  'I'actical  Inforinalion  [)istril)iU:on  Sy>icin 
list'd  iiioiithly  dociiiiii'iitcd  risk  ri'iKirts  from  iho  priiiu*  <  omrat  lor. 

Thf  Air  Force  relietl  more  on  coiiiriictors  Ilian  the  two  othi  r  se'A'ices 
did.  This  is  not  siirprisiii)j.  )>i\('n  tliat  Air  Forii'  ReHiilaiion  Th-lo  on 
source  seli'ction  calls  for  industry  to  address  risk  in  pro|K(s;tls.  Most  of 
the  12  offitt's  th.al  used  technical  risk  inforiiiation  front  contrat  tors  in 
tiieir  own  risk  I'fforls  were  in  the  \av>'  and  Air  Force.  Only  1  offite  in 
the  Army  iisetl  conirat  tor  information. 

We  ohservetl  three  firohlcms  n  ith  contractor  information  on  ti'chnical 
risk  and  risk  efforts 

1.  Contractors'  ini»tit  was  not  always  well  documented.  Sc'ven  ])ro>»rams 
obtained  information,  which  was  not  documented  at  all,  throuRli 
informal  discii.s.si!)ns  with,  contractor  staff.  When  there  was  diKumenta- 
tioii.  it  was  not  always  clear  how  contrai  tors  obtained  tlu'ir  information 
on  technical  risks.  For  example,  the  contractor  pro\  ided  technical 
reiHirts  to  the  Air  Force  rrios  proRram  office  that  included  risk  ratiiiRs 
of  a  .subsystem  but  Rave  no  explanation  of  liow  the  ratiiiRs  had  lK'<‘n 
made,  lleiue,  the  proRram  staff  had  no  oiiiKiniinity  to  evaluate  the 
information. 

2.  The  proRram  mamiRers  ir.  offict's  whosi*  risk  efforts  were  conducted 
by  tlu'ir  prime  contractors  were  limited  in  the  knowledRe  ihe\'  h:td  alxnit 
the  eflorls.  Fivi*  of  thi*  H  iiroRram  manaReis  in  thesi'  offices  could  not 
desi  rilH'.  even  in  the  most  Ri'iieral  terms,  the  analytical  approach  tlu'ir 
contractors  had  us<'\l  This  restricted  their  ability  to  iindersttind  the  limi¬ 
tations  of  the  assessments. 

d.  Some  proRram  staff  reixirted  liias  in  inforniatioii  from  industry. 
lh'S|KHidents  in  '>oiii<'  (dlices  staled  that  ti<'caiis<'  of  induslr>'’s  interest  in 
wiiiniiiR  and  niainlaiiiiiiR  contracts,  ii  ineseiiied  sy  stems  in  tin-  In-st  liRlu 
IHissihle.  i>articularly  in  risk  efforts  included  in  proposals.  I’roRraiii  staf 
re|M)rted  that  some  ratiiiRs  were  lower  than  they  should  he.  In  addition, 
tlu'y  re|Hine<l  that  contractors  left  some  risks  out  and  prolilems  iim<lei'.- 
Iified.  hecaiisi'  the  contractors  wanted  to  Ri\e  the  impression  ihat  they 
could  build  the  systems.  ( 'oiiseipient ly .  thi'  proRram  otfices  that 
received  technical  risk  mfonnalion  from  cont ractors.  es(>ecially  informa¬ 
tion  they  received  diiriiiR  souii  e  seleciion,  did  not  lK‘lie\e  I hai  this  mtor- 
malion  ai  curately  desi  rilied  a  system's  technical  risks 
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or  <-ourst“.  it  llli^^^lt  not  hnonly  l  oiilriiclors  tli;it  liad  ;in  interest  in  niuier- 
ratin^  a  system  s  te<  linieal  risks,  ixiii  in  Heneial.  and  program  tnanage- 
ment  nfCiei's  m  partienlar.  miglit  .sometimes  iia\c  been  co/isiramed  by 
the  saim*  int»-rest.  Hut  w<'  are  concerned  lieri‘  witii  the  nature  and  nse- 
fiilness  of  tt'chnical  risk  information  sniiplied  b\  contractors.  To  suptma- 
ri/.e.  tins  inbnination  wasjiot  always  widl  d(K  iimented.  lea\  ing  f)rogram 
olfic«‘S  little  or  no  opiKirtiimiy  to  gauge  its  aec  iirac>  or  monitor  (  hanges 
in  it  as  programs  progn-ssed.  (liven  the  re|>ori<‘d  bias  in  conirai  tors’  risk 
♦■ffoits,  it  is  es|Hs  ially  imjiortant  that  program  offices  Im-  able  to  e\  al- 
iiate  and  monitor  contractor  information.  Without  this  .ibility,  they 
could  Iks  <nne  overly  oplimisiic  in  making  technii  al.  schedule,  and  cost 
decisions. 


Sunmiary 


In  this  chapter,  we  havt*  idimtified  four  pioblems  th.it  stem  from  the 
services’  current  risk  efforts.  Definitions  of  technical  risk  and  risk  rat¬ 
ings  were  not  consistent.  Few  iirogram  staff  could  cite  a  txip  or  ser\  ice 
definition  of  risk  (we  disciisst'd  available  definitions  in  chainer  ’2).  nor 
could  they  say  that  any  definition  w;ls  formally  used  in  their  offices. 
.\lan.v  program  offices  used  tJefjnition.s  of  iis  hnica)  risk,  but 

these  varied  considerably  across  the  offues.  !n  (’.  of  the  2‘i  offici's,  the 
definition  varied  within  thi'  same  office.  Definitions  of  ipialitative  risk 
ratings,  whether  quantitative  or  narraliv  i-.  al.so  v  aried  w  ithin  and  across 
program  offices  and  wereofum  emit  radii  lory  as  w  ell. 

('ompleie  infoniiation  on  leclmical  risk  was  not  jirov  idevi  to  deci¬ 
sionmakers  at  the  program  management  levels  or  at  the  higher  levels  ol 
review.  While  most  (irograin  managers  wi-re  aware  of  the  characti-ristics 
ol  their  risk  effons.  some  managers  and  ot  her  stall  were  not .  The  diK  ii- 
nuMitation  and  hrielings  deserihing  lechnn  ,il  ;  isks  did  noi  present  risk 
adequately  for  the  use  of  managers  and  ot  her  reviewers. 

1’raining  in  technical  risk  asst-ssineiit  was  generall.v  lacking.  Where  risk 
was disi  uss(  (1  in  tli.>  sin  vici'  si  hools.  the  toi  n.>  was  tv  pically  on  program 
risk.  Sometimes  technical  risk  was  mimmallv  desi  rilK-d.  hut  approaches 
for  technical  risk  assessment  were  not  taiigh: 

Keliance  on  i  ontractors  for  tei  him  al  risk  mtormation  has  made  lor  sev¬ 
eral  prolilems.  Cmitrai  tors  often  performed  risk  el  foils  and  !urni'-!ted 
risk  inlormalion  tor  the  progr.im  offices,  hotli  Idinmllv  im  requests  lor 
projiosals  and  l  ontr.n  is)  and  inlormallv .  The  progr.mi  m.in.igers  staled 
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that  tht'sc  io|M»ns  may  nave  Ut-ii  hi  casi'^-  of  iiucntives  llio  con¬ 

tractors  had  to  simplify  or  minimia*  problems.  In  most  ca.sc*s,  the*  mana- 
Kors  were  jtiven  either  minimal  or  no  dcKumentation  with  which  to 
evaluate  and  monitor  a  contractor's  technical  risk  information. 
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I 


risks  ar«-  iiii  inluTfiit  imrt  of  niiijor  \v<mi>om  system  (le\  el(ip- 
menl,  imd  failure  In  iiiitu'i|>ale  these  risks  laii  lead  to  cost  and  seliediile 
prohlems  as  well  as  ilu-  failure  ol  a  s^  steifi.  The  imjHirtauee  ol  assi-ssiiij’ 
teehnieal  risk  has  lonH  In-en  reeo}iiu/.ed  iti  IKiii  and,  ar<  ordin>;l\'.  guide¬ 
lines  ;utd  refjulalions  ealiiu}!  for  these  a.ssessmenls  have  iM-en  issued.  One 
such  f’uuh'line  t  ails  for  hudfietinjf  for  teehnieal  risk,  ixtli  has  also  sup- 
l>orted  the  list*  of  tt'ehideal  risk  assessments  in  majtir  prof^ram  de<  isions. 
Itefenst'  offieials  have  told  the  Conjiress  that  only  systems  with  low  or 
nuKlerate  tet  hnit  al  risk  would  reeeive  fundinjf. 

In  this  reiMirt.  wt“  have  reviewed  the  current  stiite  of  teehnieal  risk 
assessment  iK-rformed  hv  the  Depailnient  of  Defense  lor  majiir  wea|K>n 
systems  and  attemiUetl  to  answer  six  evaluation  questions  on  jHilit  ies. 
prrK-edures.  and  applications  at  ross  the  armed  services  We  soiinht  to 
learn  how  teehnieal  risk  was  defined,  how  assessing  rns  were  designed 
and  etindiieted,  w.hal  information  was  available  to  lieeisionmakers,  and 
how  till'  results  were  eonvt'vetl  to  program  mananemt'iu  offii  e  staff  and 
milestone  revi  'wers.  Four  issues  arose  from  the  findings  of  our  inv<>sli- 
Hation.  relatinjt  to  diffieulties  in  the  areas  of  the  consistency  of  defini- 
tiotisof  risk  and  ratinf?  prtM  t'dures,  information  How,  iramint',  and  the 
involvetnent  of  eoiuraetors. 


Conclusions 


IKil)  h;is  provided  a  handbook  of  t)iiantitiiti\e  risk  assessment 
a|)proaehes  develo|H'd  by  the  Defense  Systt'tns  Manajtemeni  Collevte  in 
resixtnse  to  Initiative  1 1 .  |x)i»  has  not.  howi'xi'r.  clearly  siH-eified  its 
e.V[K'eiations  for  addressinjt  teehnieal  risks,  and  even  its  terminolojjy  idr 
eoneeptuali/.inn  risk  is  ambiguous.  There  is  no  standard  tlefinition  of 
tt'ehnu  al  risk  or  of  risk  ratintts. 

Initiative  1 1  ealletl  for  the  ,\rmy.  .Navy,  a’ld  Air  Force  totpiantify  tedi- 
nieal  risks  anti  allocate  funtls  totleal  with  them  but  has  had  a  nenliitible 
effet  t  o!.  the  ways  iht'  three  ser\  tees  handle  risk  assessment.  ( Ine  .Nav  y 
command  tried  a  '.otal  risk  assessinn  cost  estimate  jnlot  in'o^tram.  Hut 
the  .Army  simply  maintained  its  iins'xi'-tinn  Tit-VK  pmuram,  and  the  .Air 
Foret'  mainiaini'ti  its  own  cost  estimation  teehnupies  None  of  the  ser¬ 
vices  adapted  ritv:  K  or  any  oi  her  prot ediires  for  tlu  jairpose  of  quant i- 
fyinfi  and  budjtet uift  Idr  technical  risk. 

.All  2') '  roftram  manajtement  offices  uc  examined  evaluated  !<■<  him  al 
risks  in  some  way.  Ilowev  t-r.  tli\ fii  t  he  lack  of  c  larity  in  lx  ’>  del  mit  to-  s 
of  technical  risk  ami  requirements  for  tt'chnical  risk  assessmi'iit .  i  isk 
I'ffoils  \  arietl  from  office  to  td  flee  ( inly  d  ()i  o^ram  ni.inaitement  offices 
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risk  rl'lorts  (hat  wccould  classil'y  as  lis  liim  al  i  isk  asscssnii'ius; 

(hat  is.  I  line  risk  cllorts  wen- 

prosiHsjivr.  ('\aii)iiiiii>!  risks  tx-fon*  pnililcir.s  (K  i  tirrcd. 
planiusi.  iKit  ail  incidental  part  nt  prntJrani  discussions; 
exjilidt  in  attention  to  let  Imn  al  risks; 

dociimenietl.  so  that  the  '■  ‘Mills  of  an  assessnicnl  i  oiild  he  shared  \\  ith 
lit'cisioiiinakers  ainl  staff;  aii<! 

i»  oat  h  acquisition  jiliiLse.  in  order  to  tieleriiiine 

Itow  risks  ehannetl. 

As  we  have  noted  in  the  refKirt.  tht'se  criteria  are  not  necessarily  tlefini- 
tive.  blit  we  believe  they  represent  a  niiniinal  siantiard  of Hiiality  for 
risk  efftirts  m  ikid.  Kisk  efforts  in  d  prot^rani  offices  met  these  criteria, 
sitpjxiidinn  tmr  jMisititin  that  the  criteria  ale  relevant  atut  aiiainahle. 

Tiirninn  from  dt'sifjn  to  iniplemeiitation.  we  foinnl  fen  risk  efforts  car- 
rietl  out  in  ways  likely  tt>  jirodiice  the  most  ai  t  urate  and  iiscfu!  results. 
Few  provitietl  narrative  information  as  well  its  risk  ratin^ts.  coxiTcd  all 
subsystems,  or  follectctl  data  from  indt'iH-ntlcnl  raters.  Sim  i*  the  selec¬ 
tion  of  risk  a.ssessment  format,  scoih'.  and  input  prm cdure  deiH’itds 
partly  on  (he  maturity  and  comi>le\ity  oI  weaixni  systems,  tiicre  is  no 
sinule  correct  way  to  imiilement  a  risk  effon.  Ihit  few  protiram  offices 
re|Kii1t'd  lailorinjt  their  risk  effons  to  the  systems  Ix-iii);  develojx'd. 

Ihsk  rations  were  frequently  nqiorted  in  re\  lew  d<K  umenis  and  brief- 
inns.  hut  the  analytical  apiiroach  and  sco|k‘  of  the  risk  elforts  that  pro¬ 
duced  these  ratiiin^’  were  almost  ne\er  reixuied.  and  ih(‘  ratmns  seldom 
prm  ided  iniormation  on  both  the  content  and  the  level  of  risk. 

We  have  noted  that  our  study  was  not  desinned  to  measure  thei  ffect  of 
tm'hnical  risk  assessment  on  outcomes  sui  h  as  pronram  rest riicturinn  or 
cost  nrowth.  but  the  likelihmxl  of  fiiidinn  such  eftects  is  probably  low. 
The  res|>onse  of  the  Army.  Navy,  and  Air  Force  to  Iniliatii  e  I  I  was  nun- 
inuil.  and  none  of  the  'J')  jironram  offices  in  this  study  u.scd  a  technical 
risk  asse.ssment  tos';j)|Kirt  risk  budnetinn  .\Ioreo\  er.  \er.v  tew  risk 
effoils  met  the  minimal  criteria  we  de\  i‘|o|xxi  lor  e\  ab.iaiinn  technical 
risk  assi’ssments.  and  few  were  implemented  in  wa.v  s  that  are.  m  nen- 
eral,  likel\'  to  pnxiuce  the  most  u.-.eful  and  accurate  results.  'I'tius.  w  tub' 
IxKi  has  encouraned  the  assessment  of  lechnn  al  risk  and  i)ro|>osed  \  ar- 
ioiis  anal.s'lical  aiijiroaches.  it  has  jiro'.  iued  no  xtuidebnes  to  pioitram 
nianatteinent  ol  fin’s  ot  '.ow  to  iK'i  form  lei  bnical  risk  assessnicnl .  Kisk 
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assessors  were  left  on  their  own  to  decide  how  !()  carry  out  this  iinixir- 
tan'i  function.  Their  efforts  to  assess  risk  were  [hhh  ly  designed  and 
implemented,  and  the  information  available  to  dec  isionmakers  from  pro¬ 
gram  dcKuments  and  briefings  was  inadequate. 

Our  review  [xrinted  to  four  additional  problems.  First,  informal  defini¬ 
tions  of  risk  and  risk  rating  categories  were  inconsistent.  Some  pn  gram 
management  offices  had  develojK'd  their  own  definitions  of  technical 
risk  but  staff  definitions  varied  widely,  both  within  and  across  the 
office's.  Many  offices  used  qualitative  ratings  of  tc'chnical  risk  (suc  h  as 
"high,”  "mcKlerate,"  and  "low”),  but  the  meanings  of  these*  terms  were 
inconsistent,  or  contradictory,  when  examined  across  the  offices. 

Sc'cond,  technical  risk  information  was  not  always  adequately  contcyed 
to  dc'cisionmakers  and  staff  within  the  program  office's  and  at  higher 
levcl.s  of  review*.  Some  jinigram  manageme'nt  staff  members  were  una¬ 
ware  of  the  risk  efforts  that  had  bevn  carried  out  for  their  systems,  and 
others  lacked  important  information  on  the  as.ses.sment  iirtK-edurc's  and 
results.  Program  documentation  'and  briefings  often  d’d  not  provide  suf¬ 
ficient  background  on  asst*ssment  j>nxe*duri*s  or  explain  risk  ratings. 

Third,  the  training  that  isgivem  in  suiijxni  of  the  ix'rformance  of  tech¬ 
nical  risk  assc'ssme'r.ts  is  insufficient.  The  se*rvice  .scIkkiIs  cover  te'c  hnical 
risk  assessment  minimally,  and  students  are  not  provided  w  ith  the 
opixirtunity  to  practice  and  compare  applications  of  different  a,s,s<*ss- 
ment  te’chniqiu*s. 

Fourth,  the  programs  oft«*n  relied  on  contrac  tors  to  identify  tc'chnie  al 
risks  but  rcH  i'ived  inadecpiate  information  on  the  contractors'  risk 
efforts.  The  program  management  offici-s  usually  receive-d  only  the  con¬ 
tractors'  risk  ratings  and  did  not  know  how  the  risk  efforts  had  Ixvn 
conducted  or  how  the  ratings  were  defined  I’rogram  management  staff 
also  lH‘lie\'ed  that  the  risk  efforts  of  eontrae  tot's  may  have  been  buevd 
Ix'cause  industry  did  not  want  estimates  of  <*.\treme  risk  tojeouardize 
winning  and  maintaining  contracts.  (The  same  bias  may  have  affec  ted 
c'stimates  of  risk  within  the  program  offie-es  or  ixili.  bec  ause*  liefi'nv' 
officials  re{Mii1e*dly  pi«*f<*r  to  fund  systi*ms  w  ith  only  leiw  or  moderate* 
te'chnical  risk  )  The*  program  offiee-s  did  not  rec five*  siiffie  lent  informa¬ 
tion,  in  most  mst;uie'e*s,  to  e'valuale*  the*  adeepiacy  or  ace  urac\'  of  the*  eon- 
tractors'  risk  e'fforts 
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FJiiis  and  orror  arc  always  p<>ssil)l(‘  in  risk  iissessmcnt,  regardless  of  who 
pi'rforms  it.  Ikil  bias  and  error  can  more  easily  In*  uncovered  and  cor¬ 
rected  if  key  conec-pts  in  risk  assessment  are  defined  consistently  and  if 
as.>''ssment  prcKedures  and  results  aR‘  opem  to  subsequent  review. 


Reconiniendalions  to 
the  Secretary  of 
Defense 


We  rc'commend  that  the  s<H.rc‘tary  of,Dt*fen.se  take  the  following  actions 
to  imfirove  technical  risk  assc'ssment  concepts  and  procedures; 

1 .  define  technical  risk  and  categories  for  rating  risk; 

2.  require  that  risk  efforts  focus  explicitly  on  tc'chnical  risk  and  be  pro- 
spc'ctive.  planned,  and  reix'ated  at  least  twice,  early  and  late,  in  each 
acquisition  phase*; 

3.  require  program  management  offices  to  document  their  risk  assc^ss- 
ment  prcKcdiires  and  rc'sults; 

4.  eslablisli  guideline's  re'garding  optioas  for  format  for  ruling  risks, 
scofie,  data  colle'ction,  and  assc'ssment  approaches; 

5.  re'quire  that  the  tes  hrical  risk  information  that  program  offices  or 
contractors  provide  for  review  include  a  description  of  format,  scojh.*, 
data  collection,  source's  of  risk  information,  and  assessment  approaclu's; 
and 


G.  provide  more  fiK  Use’d  training  in  tcH-hnical  risk  assessment. 

Since  a  few  program  offices  have  already  perfomusl  risk  efforts  that 
met  our  five  crileri.i  .ind  since  they  have  implemented  their  efforts  in 
ways  that  are  the  most  hke*Iy  to  generate  useful  results,  it  is  clear  that 
these  recommendations  can  Ik*  iollow(*d  without  incurring  new  or  signif¬ 
icant  costs.  .Moreover,  iKin  h;is  ass<‘rt(*d  that  technical  risk  assc's-sments 
can  significantly  reduce  ci  st  growth  in  aiquiring  new  weajMin  systems. 
'I'hiis,  it  seems  reasonable  toexjK'ct  substantial  s.ivinvs  from  improve¬ 
ments  in  the  design  and  imp'ementation  of  these*  assi'ssment.s.  Of  course, 
our  reconimend.'ttions  concert,  only  oni*  elenicnt  program  m.inagcnient 
and,  by  themsehes.  cannot  ensure  timely  and  efficient  dcM'lopment 
effoils 
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(  4>nrliui»i4>ii*  and  Krci»miiM‘iKiji|j4m«i 


\tfcncy  C'OIllIllCIltS  <in(l  ’‘‘"''‘l  •*  ''f  tl'i*'  rf{x>ri.  IX)|)'S  cDamicnts  and  our  I'omiilcto 

-j  p  .  rrsjwmsi- aiv  m  upix-iulix  Hi,  ixiii  ^fuorally  comiirrod  wiih  the  pnunpal 

Jiir  liCS^yollSG  fimUn^’s  inif  ar>;iio«i  lliat  ilu-  rcjnirt  ovcrcmpluLsizos  tt‘<  Jiiii<  al  {jrohlcnis 

ii-s  (iistiiu  I  fr<im  tho  cosi  ami  m  lu'dulf  corujxun'iUs  of  overall  program 
risk,  uip  fonturrixi  fully  or  partially  with  all  riToinmendations  I'xtrpl 
the  one  l  allinj;  for  making  additional  inforniatioti  on  risk  a.s.s<'.s.sniciit 
profixiurcs  av  ailable  for  revu'w  (it.vo’s  fifth  reconinicndation).  ixiii 
expressed  relui  taiue  to  plave  further  rerjuireiuenls  on  projiram  mana>te- 
luent  and  arjiuevi  that  eost  growth  (uls  deeiined  to  aUnil  1  ix  rcent,  ren- 
derm^t  .siiefi  requin'nients  unmx-essary. 

We  iK'lieve  that  the  findintfs  demonstrate  a  nix'd  for  more  i  larity  in.  and 
attention  to,  tix  linieal  risk  avst'ssmenl  in  ixiD.  'I'iie  findiiif’s  do  not  sutj- 
t;est  that  terhiniea!  risk  is  nion>  eriiieal  than  eost  risk  or  schedule  risk  or 
that  iKUi's  attention  to  cost  or  schedule  risk  can  be  ri'ilueed.  We  have 
mxintriu-nded  more  eonsisleney  in  iLssessment  concepf.s  and  pnv'ediires. 
but  we  also  reaiftnize  tin*  luxxl  for  tailorinj;  assessmcnls  to  particular 
Iirojtrams.  Since  most  <>;'  tx'h’s  Jtsvssments  did  not  mirt  minimal  stan¬ 
dards  of  quality,  it  is  unlikely  that  they  iiace  contribute  J  to  any  rcxiuc- 
tions  in  cost  >>rowth. 
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rograiTi  Descriptions 


I 


This  aptK'iniix  hi  iofly  dcs<.ril)<*s  <-;Kh  program,  il.s  iiiH  iHiifl  juiriHiM-.  and 
tiio  effort  to  identify  its  le<  hni(  al  risks.  For  some  pronrams,  effo'is  were 
formal  an<l  discrete  tasks.  For  others,  they  were  informally  jiart  of  pro¬ 
gram  office  routine.  Many  risk  efforts  were,  in  some  res|K>i  is,  extensive 
and  carefully  dime.  Seven  of  them  met  four  of  the  criteria  we  develoi-  d 
for  this  evaluation.  But  only  threr*  risk  efforts — for  the  .Antisuhniarinc 
Warfare  Standoff  Wea|)on,  Remotely  I’ilot«l  \'ehicle.  and  Short  Ram'e 
Air  Di'fen.se  Oimmand  and  Control  System — met  all  five  ••svntial  cri¬ 
teria.  Risk  efforts  for  all  programs  are  evaluated  m  terms  of  the  crili  ria 
in  table  3.1 . 


The  Army  Helicopter  Improvement  Program  (aiiii')  s(s-ks  to  iipgraile  the 
capabilities  of  the  light  obMTvation  heliaipter  fket.  The  de\  elopment 
effort,  contracted  to  Bell  Helicopter  Textron,  covers  1-1  subsystems, 
among  which  are  a  target  ohsc-rvation  and  acguisition  devui-  tilKive  the 
rotor  (a  mast-mounted  sight),  the  tail  rotor  drive  shaft,  ami  navigation 
and  comm-T.'cntmn  eqo’pr'.ent.  ftm*  is  shiieii  to  handle  reconnaissam  e 
security,  and  target  designation  and  handoff  in  supiiort  of  attack  heli- 
aipters,  air  cavalry,  and  field  artillery,  it  is  exins  t»’d  to  oin-rate  day  and 
night,  in  hot  weather,  and  at  nap-<if-the-canh  altitudes. 

TTie  Army  eypres,s<‘d  interest  in  an  advanml  scout  helicojiter  in  IB7  I 
but  decided  .a  years  later  that  an  entirely  new  helicopter  svas  not  afh  rd- 
able.  In  HtHh,  tlu*  Army  began  planning  for  a  scout  helicopter  that  would 
Isilster  the  capabihtu*s  of  au  existing  miHlcl.  Full  si  tile  enguusTing 
development  for  Aliir  started  in  lU.m,  under  the  direct  ton  of  the  .Avia¬ 
tion  .Systems  Command  in.st  J<oui.s.  .\!is.v)uri  Formal  Popievirw  for 
milestone  H  was  in  i-arly  JBH2. 

In  IPHl.  a  division  risk  luialysis  wtis  jx-rforimsl,  in  preparation  for 
source  si'lciiion  for  the  development  contract.  In  (x  rsonal  interviews 
and  a  written  nuestionnaire.  tis  hnn  al  and  engiius-nng  stalf  laled  risk 
for  each  AtUI’  subsystem  on  a  six-p-oint  scali-  defined  m  (jtialilaliv  e  leriiis 
ranging  from  "none  or  verv  low”  risk  to  ■  iinaccepiahly  high"  risk.  Tlie 
(jui-stionnain'  jirovidisl  a  •.erhal  desi  riptiou  of  each  (lotni  on  the  s<  .ile 
and  of  lower  and  upjH-r  Ivmndanes  for  the  prohahi'  ty  of  not  meeiing 
|n’rforman<  e  nsunrements.  For  example,  "umu  <  epial'lv  high"  was 
descnlH'd  as  '  concept u.ili/.eii  on  paji-r  but  •'till  tlixiii  ti.  a!  .md  m.t'. 
exceed  current  stai.-of  the  ail.”  In  quantiialive  terms,  risk  w.is  '  iin.ii 
o-ptably  high"  if  the  profi.it>ilify  of  not  im-eling  reqiiireinenis  e\ris'd<sj 
riO  jH-rceiit.  A  support  stalf  mi-mlx-r  siimm.u  i/ed  the  ratings  .md  lh.  it 
iisi'd  them  as  mp'.il  for  a  compuleri/.»-d  schedule  risk  ana!\  ^ts.  w  hu  h 
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ficiicriitcd  \  ;in(ms  rst  iiu;iti's  of  I  imc  to  foiiiplrt  loii,  sir  h  ;is  |n'i  rent 
('liamo  of  lotiipU'doii  withm  .'<7  months.  hO  jK-m-nt  ^^  itlnn  ;i;i  niotiths  ” 


The  Navy  Mark  *>()  Advanced  l.ij^ht  weight  T(ir[H-(lo  (At.w  i')  is  an  antisuh- 
inanne  tor|>edo  ciesifinc’d  to  enhance  eapahilities  for  tar>;et  ai  'jinsition. 
stH-vd,  lethality,  and  depth.  Its  sonar  s\  stem  is  mtcaulc-d  to  d<’tc-c  t  targets 
fiLsier  and  in  f^reater  volume's  of  water  than  earlier  torjx-dirs  could.  Its 
eny.me  is  intended  to  render  the  tor;H‘do  faster,  quic-ter.  and  ahh-  to  dive 
dcs-|H*r  than  conventional  eiiHines 

I  'nder  the  Na'. al  .Sc-a  Systems  Command  in  Cr\stai  '^’ily,  \■ir^;lma.  and 
under  contract  to  Honeywell,  the  ai>^'T  passed  milestone  II  for  full-scale 
devc-lopment  in  c-arly  lhH-1.  It  is  sc-l  for  a  ,  rcKliiction  dcs  tsion  (milestone 
III )  m  late  lilSd. 

The  Al.WT  IS  a  pilot  projtram  for  the*  Na\  al  M.iterial  Command  risk  man- 
aHemc-nt  system  called  '  solvint;  the  risk  csjuation  in  transitionin>t  from 
devc'lopmeiit  to  prcKlueticm."  The  pro>;ram  offits-  has  or^ani/.isl  its  risk 
management  to  conform  to  the  command's  j’uiclelinc's  Kviensn  e 
monthly  and  himonthly  rc'(K>rts  f’-om  the  eontraclor  have  piov  uted  <  nr- 
rent  iiro>;ram  dal-c.  suc  h  as  tc-st  n-sult-s  showmn  the*  "mean  time  h"!  aism 
failure"  for  various  aI-WT  cor..{H>nents,  Kesiilt.s  lun  e  Iss-n  a>j^re^;aled  in 
various  \\a.\s  to  tc’llc-ei  tc-c  hnu  al  risk,  and  hi^th-risk  com|Hiiients  have 
Ins-n  clivusscsl  in  nus  tiiifts  iK'tvccs’n  pro^tram  office  stafi  anil  the’ 
c  on  I  rad  or. 

Sirne  memts-rs  of  the'  staff  dcs  u|ecl  to  suppli'iiient  the  cominand's  Hiiidc'- 
linc's  with  an  addiIion.il  mea.surc' i-.*’  risk  not  t)as«-d  on  lest  results,  TJ.eir 
measure.  iUKlaled  monthly,  ratc-s  risk  for  eai  h  .MicT  suhs\  stem  on  a  one- 
to-live  sc  .de  It  h.is  Ims-ii  inc  liidisl  in  t'>e  eonlr.n  tor's  re|M»it.s 
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riie  ,\ir  I'one  .\d\  am  isl  'Icslnim  Kanite  .-\ir  to  ,\ir  Missile  (  wiitv  vm  )  ;s 
an  ;d!  \ve;iih(-r.  radar-Kmd'-d  missiii-  dc'sij.'ned  for  .-\ir  Ton  <■  .iml  .\a\> 
fitthter  ain  ralt  (  ompan  cl  to  missilc-s  c  ur  n-iii  !\  m  proclin  t  ion,  amicv  vm 
will  n'lsirledlc  tx'  Ic-ss  de|x-ndeni  CM  Us  l.iiim  t'.ra;  plat  till  m  l>>r  I  ai  >tet 
d>  •avln.ii  ,'Mii  am)  eindaia  e  It  will  he  >;ii;dt  c)  to  Ui<-  .urei.cli  r.td.ir  until 
111 II ii  oi It  >e,  ■»> ! ,.  ii  It  Vi  :1!  Ui  t It  It s  ciw  1!  ;  ,11 I  !  ;  !  a ii’.i  (i  .un M-  •  i  e 

e.ip.il 'lilt >  is  miended  lo  .dlow  the  jnliii  to  h.'‘i'.ik  atc.i>  .do  r  firip^-iul 
eii^a^.  Ill  her  Ian'.  Is  I  Udi-r  de\ el' i|.iin  nt  l'\  Ihiidies  .\in  t  .it^^vu.S  \'t  is 
|x|ll|(  ili-'U'.nc'd  .d'Mi  1 1  >1  v'.i  I  .Ui-|  s(»-ed,  I  ell.  1 1  ;  1  it  v  ,  .eiil  I  esi  <'  to  '  lei 

iMiiiii  I  l•im^enlll  .i-.iiiex  :|.,in  imsstlc-s  now  ,iri«tr,  -  I 
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Air  Forci'  Base,  Florida,  amic\am  pasM-d  miU-slonc  li  for  fnll  sralc  dr\  i‘l- 
'ipmcnl  in  IftS'd. 

Tho  primary  risk  (‘'.Tori  was  hamikai  ky  ongoinjj  [irojjr  am  adiviiics  sut  Ji 
as  rcHiilar  inoctiny's  of  tlic  program  mana^'cnu'ii'  staff  and  c  (intrador  lo 
discuss  test  results  and  identify  the  jiro^tram's  ■  tcchr.ical  drivers."  This 
approacli  led  to  efforts  to  reduce  risk  that  were  renis  ieti  in  < ontraet 
siK'cifications.  comiietition  hetvvi'eii  contractors  diirinj;  tin*  dimionstra- 
tion  and  validiition  phase,  and  projtrain  restria  tiirinr’. 

Other  risk-related  activities  h  i  amicvam  tire  contractor  reixirts,  cost  and 
schedule  an;ilys(‘s.  and  a  riTCU  study  of  tlu*  overall  program  hy  a  hlue 
ribbon  panel  of  Air  Force  and  Navy  rev  iewers. 


The  Air  Force  Antisatellite  \\’eaiHm  (.as\T)  is  designed  to  destroy  sin'ci- 
fied  low-altitude  satellites.  The  .v.\\T  wetijHin  comprises  a  two  staKe  mis- 
.silo  and  a  miniature  homin^J  vehu  k'.  The  .as.\t  is  to  be  la  ineht'd  from  an 
F-lo  finhter  jilane  into  sptu  e.  w  lu'fe  the  minuiture  homing  veluele  would 
maru'tivt'r  into  a  salellit(‘'.s  orbit  and  de.siroy  it  by  (iireei  iminu  t. 

The  ASAl  is  beiiiH  develo|x*d  by  the  Air  F  ),  ee  Space  Division  in  F.l 
Scfiiindo.  C'tilifornia.  Boeing;  Ai'rosp.ict'  ('oiniiany  is  the  eontrtu  lor 
res|H)iisi!)lr  for  the  missile  and  system  integration;  the  minititure  vehicle 
is  being  built  bv  id’V  Aerospace  an.l  I 'crease  Company  The  system  has 
been  under  tieeel'Ttited  (levelopmeni  tind,  w  hen  we  fmish"d  datti  collee- 
tion,  had  not  yet  ban  any  fc.  inal  nwitc  mik'stone  review.s. 

'I'he  primaiw  AS.vr  risk  effort  was  ;v>rformed  hy  the  program  office  lo 
meet  th.e  information  n-reds  of  authorities  at  higher  levels.  A  prohahdity 
of  success  for  a  system  test  was  i  omputed  hy  co.nhiriing  prohahilities  of 
success  for  the  iK'rforiuanee  of  eaeli  suhsvstem  Qu.ihlalive  ratings  of 
the  level  of  risk  '  high,  medium,  'ovv  )  were  iCvSigned  to  i  aeh  aiea  of  tecii- 
meal  eoneern 

The  program  office  also  had  .iddilional  information  on  ici  hnic.il  rii^ks, 
develoix'd  through  inform, il  assessments  i>erforme<l  (luanei i\  for  the 
vieetc'd  aniinsit ion  report  and  program  rev  i’  vv  These  (ssess-neiils 
relied  on  engineering  judgmiaU  for  sulijectivi  estimates  (T  the  i.s  him  al 
risks  of  the  system  ( n her  rr  k  miormai  ion  im  hided  foi  lual  t  o'  l  i  isk 
estimates  lejtorted  hy  the  eontraelor  for  the  mim.iture  vchu  le 


I'<«r  N2 


CiAO  nicul  Risk 


AppeiKiix  I 
Prt>iCrani 


ASFJ 


Tiic  Airbcnu'  S«‘lf-Proli-t'ti()ii  JamnuT  i.-.  <ii>  v 

list'd  u)  provide  lai  tica!  aircraft  witli  the  cajialiilily  for  defensive  elec¬ 
tronic  counterinciLsures.  It  is  desi>;ned  to  fit  aboard  a  variety  of  aircraft, 
includin}*  the  .A-d,  AV-8B,  F-14,  F-lfi,  and  I’-18  The  systi'm  is  desiftni'd  a.s 
five  modules  to  allow  different  installation  configurations  to  meet  the 
rt'quirements  of  individual  aircraft. 

The  Navy  is  functioning  as  the  lead  la'rvice  in  this  joint  Navy  and  Air 
Force  pronram.  Management  res,K)nsibilities  are  under  the  Naval  Air 
Systems  Command  in  Crystal  City,  \’ir)tinia.  The  proftrtim  began  full- 
scale  developmeni  after  passing  mili'stonc  II  in  August  H)7!).  I'he  jtro- 
duction  decision,  milestone  111,  is  sche  iUlcd  for  1 8S(i.  The  system  h;is 
been  jointly  ilesigned  and  dovelo|K‘d  by  ITT  A'.  ionics  and  W'e-.i  ingtiousi' 
Defense',  but  the  team  membt'is  will  be  required  to  conqx-te  for  the  pre>- 
duction  pluLse  of  the  program. 

The  program  office  has  pc'rformed  ongoing  risk  manage-ment  and  risk 
rt'duction  e'fforts,  reacting  to  probk'ins  as  lhe>  arise.  Test  ri'sults  hav  e 
bevn  I'.'lie'd  on  to  revi'al  areas  ri-quiring  attention. 

Asse'ssments  of  program  cost,  schedule,  and  technical  risks  were  carrii'd 
out  by  a  supp.*>n  contractor  when  this  program  was  pan  of  the  pilot 
total  risk  asse'ssing  cost  estimate  (Tlcv'K)  program.  According  to  program 
jx'rsonnel.  the  TtevK  funding  for  the  iirograin  was  cut  from  th-  budget 
and  the  program  is  no  longer  part  of  the  pilot  effort. 


ASW  SOW 


Till'  .Navy's  Antisubmarine  Warfare  Standoff  Wc'aiKin  ^.^.sw  s<m  )  is  a  sub¬ 
marine-launched  missile  designed  for  quiet,  buoyant  asci'iit  and  short 
"time  to  target."  It  is  a  single-stage,  rocket-proix'lled  missili'  with  two 
payload  alternatives:  the  nuclear  depth  bomb  and  the  advanced  light- 
wi'ight  lorjK'do.  ft  is  intended  ;ls  a  t  dctical  antisubmarine  weaiHin  for  the 
txS.N-(>.‘J7,  SSN-()88,  and  follow-on  submarines.  The  program  is  run  by  the 
.Naval  .Si'a  Systems  Command  in  Crysttil  City,  \'irginia. 

In  February  IPHU.  four  comiianies  weri'  awarded  contrai  ts  for  a  concept 
formulation  sfidy  of  the  .\.sw  si  w.  From  the  results  of  these  studies  and 
the  i)rotx;sals  each  company  submitted.  IkH'ing  Ai'rospace  was  cliosen 
for  the  demonsir  ition  and  v  alidation  work  on  the  system.  The  program 
office  received  milestone  1  apiiroval  in  December  1PK2  and  [ilans  mile¬ 
stone  II  for  .June  188(5. 
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I'hc  primary  Asw  s<M'  risk  I'ffort  vsiLs  jHTf<inm‘d  t)y  ii<)c-inf». 

1  i-^k  was  assessed  as  part  of  the  risk  ma:;;>.i!,emenl  effort  required  m 
Ikainft's  eoiUract.  Ikieinj;  identified  eijtht  areas  (  f  technical  n.  k  ann  lias 
continued  to  monitor  these  areas  in  the  di'inonst ration  and  va'  dation 
phase.  Three  main  activities  were  jaTformed  in  order  to  idei  ify  risks. 
First,  a  “factory-to-tar>’et  sequ-'iice"  malri.x  was  develoiK'd.  laying  out 
the  acquisition  steps  from  conqx-iicnt  fabrication  to  launch  for  each 
work  breakdown  structure  element.  Si>*nificant  events  in  development 
and  environmental  considerations  could  Ix'  taken  into  account  by  using 
this  matrix.  Ratings  of  high,  nuxlium,  and  low  were  given  to  the  ele¬ 
ments  with  risk,  fk'cond,  a  risk  element  matrix  was  develofied,  majiping 
the  work  breakdown  struc  ture  itcmis  against  what  Ikx'ing  calls  risk  ele¬ 
ments  of  cost,  schedule,  ix-rformance,  reliability  and  mainlaiiiabiliiy, 
prcxluction,  and  safedy.  Again,  high,  medium,  and  low  rat  inf’s  were 
assigned,  as  dc'emed  approjiriate.  Third,  bexause  certain  items  tc>nd  con¬ 
sistently  to  cause  problems  in  .system  development,  data  from  other 
Ikxdng  .systems  were  used  to  identify  risks.  Ikx-ing  regularly  reviews  the 
system  for  potential  risks  other  than  the  eight  tliat  were  found  from 
these  thrw  activities. 


Risks  are  asses.sed  and  monitorc'd  by  a  risk  man.igement  Ixiard,  a  small 
group  of  Ikxdng’s  asw  scxv  management  ixTsonnel.  The  Navy  S<>a  Sys¬ 
tems  Command  technical  refirc'sentative  at  BiHsng  is  invitim  to  the 
formal  meetings  and  rc'ceives  a  copy  of  the  niiiuUi's  The  risk  effort  and 
the  standards  for  rating  risk  have  Ixvn  diH  umented  in  IkKung's  risk 
management  jilan.  JkK’ing  has  also  documented  the  effect  the  risks  are 
expected  to  liave  on  the  firogram  and  the  stejis  that  will  Ix'  taki-n  to 
abate  them. 


The  Navy  Advancc’d  Tactical  Radar  .System  (  .m.';.-')  is  an  ami  air-w  arfare 
system  to  lx*  u.sed  in  sufiiKirt  of  the  defense  of  local  areas,  fhe  .Na\ >■  is 
still  defining  the  atks  conceiit.  but.  generally,  it  has  been  jilanned  as  a 
system  that  will  have  both  a  surveillance  and  a  weafsin  sup|x)rt  func¬ 
tion.  It  is  tx'ing  dc'signed  for  several  iilatfornis,  including  the  ne.xi  gem-r¬ 
ation  of  surface  shiji  combatants. 

The  ATRs  Jiad  its  genesis  in  IhSli,  and  the  ojx-rational  re(|uirements  wc-re 
dcK  unientc-d  in  .Ianuar\'  IbKl  Status  ;ls  a  major  system  was  arbu-\ed  m 
S<‘i)temlK'r  HIS-1.  The  program,  being  dc‘\eloi)c‘d  under  the  ,\a\  .il  S-a 
Systems  Command  in  Cr  v  stal  Cit\-.  \  irginia,  has  rei  iained  unfended 
du.'inga  reevaliialion  of  the  reiiuiremeiiis  .\  milestone  11  rev  u-w  is 
»-X)x-cled  late  in  fiscal  year  IPSS  or  early  m  fiscal  year  IPS') 
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A  siH^ial  roiisistinn  ol  staff  from  tlie  systems  commaiul.  sop[>orl 

laboratories,  the  program  management  office,  and  others  is  heljjing 
define  the  ATKS.  The  program  manager  has  .said  that,  because  it  is  so 
early  in  the  acquisition  cycle,  the  risk  effort  has  been  limited  to  informal 
discussions  of  areas  that  may  contain  risk.  The  desi  'n  options  presented 
for  the  system  were  the  inuietus  fo  these  discussions.  The  program 
manager  planned  to  use  qu.intitative  risk  efforts  for  reporting  to  pro¬ 
gram  manage  ."lent  and  nu  i(>v\’er.s. 


Thec-i7A  Airlift  Aircraft  System  will  be*  designed  to  fierform  a  full 
range  of  airlift  missions  in  interthealer  and  intrathcater  roles,  inc'iiding 
air  drops,  combat  oflload,  mt'dical  evacuation,  and  low  and  normal  alti¬ 
tude  parachute  extraction  of  various  types  and  sizes  of  cargo.  It  is 
intended  to  deliver  cargo  into  small,  austere  airfields.  The  t-i7A  will  be  a 
turbofan  wide-body  aircraft  pow  ered  by  four  engines  bt'ing  certified  by 
the  F'ederat  Aviation  .\dministrat.'o:'i  for  commercial  aircraft.  It  is 
intended  to  replace  the  active  fic-et  of  C-141B  aircraft;  it  may  also  be 
used  for  roles  currently  filled  by  older  C-I30  aircraft. 

The  ('-I7A  was  initiated  in  1!I79  (known  then  as  the  C-X)  under  the  Aero¬ 
nautical  Systems  Division  at  Wright-Pattersem  Air  PoK-e  Bicse,  Ohio.  In 
.lul/  1982,  the  Air  Force  awarded  a  contract  to  .McDonnell  Doughts  Cor- 
jKiration  for  a  iiv'destly  jiaced  C-17  research  and  development  program, 
and  this  received  milestone  11  approval  from  the  Air  Force  Systems 
Acquisition  Heview  Council  in  1981  and  from  the  ps.\Kf  in  November 
1984.  A  mik*stone  III  review  is  planned  for  fi.scal  year  1987. 

The  risk  effort  has  Ik'ci'.  carried  out  i.iformally  in  the  program  offii  e  as 
a  part  of  routine  management,  through  technical  interchange  meetings 
held  regularly  wit!i  tlie  confactor  to  discuss  technical  problems  and 
is.sues.  Kach  meeting  Iuls  bemi  stnictured  around  a  particular  fiim  tional 
area  of  the  phme.  .so  that  different  subsystems  are  examined  at  dil  fercnt 
nu'etings. 

Technical  risks  for  .imi  were  also  examined  during  source  selec¬ 
tion  I 'ndi'i'  Air  •gulation  7(1- 1  .o,  offerors  were  required  to 

address  ler  hnieal  risks  in  their  proiKisals,  and  tlie  source  selection  e\  al- 
nation  Ixiard  considered  the  risks  m  sell  rting  tin-  winner 


The  i'\  Inner/.oiU'  Antisubmarine  Warfare  Ih  hcopter  ((  v  iiKl/i)  was  initi¬ 
ated  to  provide  a  r  apahilitx  for  fast  reai  tion,  higlil\  mol.'ile,  ai  li\e  sonar 
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and  tnrjx'di)  drlivory  to  a-isi;--  in  d(*t('c  linn,  liK-atiiiK.  and  altac  kin^; 
cncray  submarines  entering  tin-  liijjli  ’ioisc  vr. .  ironinciU  of  ilic  t  an  icr 
battle  >;n)U|)  inner  zone  It  is  intended  to  re[  !aee  theSll-3H  belie()j)ter 
eurrently  in  sor\  ice. 

r)evfJo|K‘d  tinder  the  f^ukiunco  nf  Nava)  Air  Systenrs  f’oininand  in 
Co'stal  ('ity.  Virginia,  the  t  v-iiKlx)  pitssed  a  miler  lone  11  review  in  .lan- 
uary  iftHo.  y\  eonlraet  for  development  wivs  subseqiieiuiy  awarded  to 
the  Sikorsky  Aircraft  Company. 

The  projtram  office  hits  examined  and  reivirted  tts  hnical  risk  issues  in 
the  pro}*rant’s  monthly  stalvis  rejx.rt.  Ti'chnical  risks  have  tn-en 
addressed  subjectively  in  informal  discussions  witf;  the  [iroftram  s  engi- 
neerintt  [lersonnel. 


HFAJ 


The  High  Frequency  Anti-Jammer  (llKAJ)  is  beinR  develofH'd  to  provide 
anti-jam  protection  for  tactical  battle  grouii  ojx-ratioas.  The  iiK\J  uses 
broadband  frequency  and  has  the  ability  to  hop  in  the  hitth-frequency 
spectrum.  It  isex|H'cled  to  provide  a  system  witli  better  availability, 
automation,  and  efficiency  than  the  system  currently  used.  The  five 
parts  of  the  system  are  the  exciter,  rei  idver,  broad';  nd  ixiwer  amjili- 
fier,  anti-jam  modem,  and  anti-jam  controller. 

In  June  1081 ,  the  Chief  of  Naval  Oix't  ations  ajiproved  I1K/\j  develoi>- 
ment.  The  Navy  subs<>quently  awai  ded  contracts  to  XiK-kwell-Collins, 
Westinghouse,  and  (iTB  for  advanc  d  development.  The  program  office, 
under  the  Nava!  Flectninic  Systems  Command  in  Crystal  City,  Virginia, 
was  workiiif*  toward  a  milestone  II  decision  in  1084  when  the  sran-tary 
of  the  Navy  stopiR-d  the  fundim;.  .Since  then,  the  system  has  Iteen  under 
re  .'iew. 

The  primary  tiKA.i  risk  effort  has  laon  condiicttsl  by  priittram  manat;'-- 
ment.  At  meetings,  risk  is  discussed  in  an  informal,  r.ubjectix'c  approach. 
Test  results,  w<;rk  on  other  systems,  jH.'r.sonal  exixTience,  and  the  opin¬ 
ions  tif  •  nginisTS  and  laboratory  scitmtists.  amont!  other  tliintfs,  have 
bi'en  CO  isidi-retl. 


I-S/A  AMPE 


The  liiter-S<‘r\  ice/A>;ency  Automated  Messiivte  I’nKessintt  Fxt  hanUe  (i-s/ 
A  AMi’K)  will  handle  stsure  .ind  yteneral-serx  ice  eommatul,  contn.l,  t  om- 
municatioiis,  and  iiuelhitent  e  for  the  armed  services,  other  iiment 
agencies  fsiich  its  the  National  .Vciirity  Ageiu  y  and  Defense  Intelligence 
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Agfi'i  y).  Mild  r.S.  aliifs  Ovi-rall,  about  2. IKK)  umts  an-  cxiH-ctod.  It  is 
bciiijt  licsiniicd  to  modorni/.o  .1  staiidardi/o  iiiria-iit  barduaro,  soft- 
watv.  and  prooodiiros. 

Tlio  l-s/A  ANtl’K  proftrani  lias  iiiKicrUonc  sovcral  sbifts  in  eoiucpt  defini¬ 
tion  since  |>!anninn  beftan  in  l!)7r).  I'lie  Air  Font'  Ixt  anie  lead  service  in 
lb7f)  and  assijtned  the  jnojtrani  to  its  Aiilonialed  Systems  I’roftrain 
Office  at  Gunter  Air  Force  Station,  Alabama.  l-s,'A  ami’K  pa>sed  mile¬ 
stone  1  in  l!>8:j. 

The  primary  t-s/A  ami'K  risk  effort  b;is  been  conducted  ;ts  a  set  of  man- 
ajiement  [iractices  and  decisions,  including  offeror  conferentt's  and 
surveys  to  evaluate  desi)>n  alternatives,  review  by  M'rv  ice  laboratories 
and  ex(K'cted  tist'rs,  indeiH*ndent  validation  and  verification  of  ti*chnical 
plans,  required  certii'icalion  by  the  .National  Se*curity  Agency  of  each 
system  comp-ment.  tests  of  critical  comjxinents,  and  work  plaas  that 
standardi/A'  the  contractors’  t'ffort.s  and  promote  the  inte^tration  of 
(ximiKinent.-. 

As  a  result  of  activities  like  these,  proytram  management  adopted  a  two- 
track  development  stratej'y.  Track  1  is  the  development  of  low-risk 
items.  Items  not  yet  "reduced  to  practice"  will  be  added  later,  if  feiLsible, 
in  track  II  "preplanned  priMliict  improvement." 

Two  other  management  activities  were  a  1982  internal  audit  reiKirt  that 
discussc'd  technical  issues  and  an  indeix-ndent  cost  analysis  jx-rfonniHi 
in  HtHd.  A  comimterized  system  monitors  the  dexclopnient  si  hedule. 


JSTAiiS  ^  loini  Surveillance  and  I'arKv't  Atlai  k  Hadar  System  (  istaks)  is 

designed  as  a  siirveillaiuc,  battle  loanaHement.  and  tarjtet  attack  control 
system  to  detect.  Icn-ate,  and  track  liir^ets.  The  .isT.Mts  Includes  C'-IH  air¬ 
craft,  airtse  lie  radar,  airl.'orne  and  ground  data-priHcssini;  and  disjilay 
I'quipment,  secure  anti-jam  voice  and  data  communication  I'qiiipnu'nt. 
ground  station  modules.  wea[)on  interface  units  alK>ard  rijiliter  aircraft 
jKiti'iitially  alile  to  c  arry  missiles,  and  software  supjHirt  ;md  develop¬ 
ment  facilities. 


The  .tSTAlts  vv  IS  initialed  as  a  joint  Army  and  .Air  Force  program,  w  ith 
the  Air  Fon  e  ,ls  tlu' «  xecutive  service,  at  Hanscom  .Air  Force  Base  in 
Ik'dford,  V.a.ssachuM'tts  The  joint  program,  fornu'd  in  M.iy  1982, 
merged  tvo  jirograms.  the  Air  Force  I’ave  Movc'r,  a  system  lor  cK'tec  ting, 
liK-ating,  acd  striking  mobili’  c'lieiny  armor,  and  the  .Army  Standoff 
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JTIDS  (Air  f'orce) 


'I'arj’ct  Atqiiisiiion  System,  a  radar  system  for  fast,  continuous,  and 
broad  helie«>pter  surveillance  of  rnovin^t  ‘tround  tar>.',<’is  Contracts  were 
awarded  toGeneral  Klectric,  Gnimman,  HnUb'  s  and  WestinUliouse  for 
studies  of  the  radar  and  antenna. 

Thrt'e  risk  activities  were  di>stTib<'d  by  the  resiK.mdents  in  our  ir.ter- 
views  in  the  proj*ram  office.  One,  was  done  solely  on  the  antenna  by 
Rome  Air  Developnumt  CetUer,  the  technical  arm  of  the  Air  For  ce  Hiec- 
tronie  Systems  Division  at  Hanscom  The  Uroup  that  a.s.sessed  risk  c<,<i- 
sisted  of  tliree  eiiKine^TS  and  a  refirestmt.ilive  of  .Mitre  Corjxiration,  the 
system's  enjfincvnng  contractor.  Qmtractors  presentcHl  to  tliis  group  the 
work  they  had  done.  Following  the  contractors'  presc'iitations.  the  group 
layed  <iut  a  matrix  describing  what  each  contractor  had  done  in  the  C;‘.;r 
areas  that  it  judgc-d  would  lx*  a  problem  in  developing  the  an cnna  and 
rated  these*  areas  as  high,  medium,  or  low  risks.  'J'he  group  hr. -fed  the 
program  director  and  Air  Force  officials  on  their  results.  The  primary 
risk  effort  dc“scribed  in  our  report  c-ompri.sed  these*  thri*e  activitic*s. 

The  program  r.ianagement  staff  have  also  dealt  with  risk.  Mexleling, 
prototi  iiing.  tee  hnical  studies,  and  engini‘c*ring  judgment  have  heljied 
the  staff  make  informal  assc'ssments  for  decisionmakeiN, 

In  accordance  with  .Air  Ftirce  Regulation  70-ir>,  risk  was  also  a.ss«-ss:.'d 
by  the  source  se'lection  evaluation  iKiard.  lk*fore  tin*  proixisals  were 
revieweil,  factors  on  which  they  were  to  In*  rated  and  standards  for  rat¬ 
ings  were  establislusl.  A  separate  higlvmedium-low  i  .it  mg  scale  was 
apt)lied  for  technical  risk. 


Tin*  .Joint  Tactical  Information  Distriliution  System  (.iniM  is  a  time-tiivl- 
sional  multiple-access  communication  system  inti'ndi'tl  for  jam-resistant 
digital  ( (immunieation  of  data  and  voici'  for  (ommand  and  control,  posi¬ 
tioning  relative  to  navigation,  and  identification.  'I'he  Air  Force  and 
Army  .fxiiis Chess  2  terminals  for  the  s.vstem  an*  designetl  for  fighter  air¬ 
craft.  ground  tactiial  vehicles,  and  installations  that  have  siiaee  and 
weight  restrictions  The  Class  '2  terminal  is  comixis* d  of  a  ten  i\er  and 
transmitter  unit  develojwtl  liy  the  Collins  Government  .\\  ioim  s  iMvision 
of  Rockwell  IntiTiiationai  and  a  data  {UiHcssor  unit  (lm.elo(H-d  tiy  the 
Kearfott  DiMsion  of  the  .Singer  Com|»,my.  Witliin  thi*  data  [iriH  essor  are 
the  interface  unit,  digital  data  prix-essor,  s«*<  ure  data  unit,  and  h.ittery 

Advanced  development  modeling  of  the  Class  2  terminal  in  ,he  lati* 
HlTli's  siipjKirletl  the  usr-  of  .iTilis  t.ii  |)latforms  v^•|lose  s[iace  i>  restricted. 
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Prognua  Description* 


Tlie  pro^jraiT  \v;ls  ai)pn)v**d  for  fiill-sc  ali' devc'lopm'TU  in  January  r)81. 
Milestone  HI  pr<KiiKtion  revi('w  is  planned  for  May  UtSti.  The  Air  force, 
as  lead  ser\  iee,  runs  the  program  from  the  hleetronies  Systems  Division 
at  Hanseom  Air  Force  Hasc  in  lt<-dford,  MassachusetLs, 

Two  risk  efforts  were  deseribc'd  in  our  pro^jram  office  intervicv.s.  The 
primary  effort,  which  wi  lejxirted  in  cliapler  3,  is  an  elei  ’ent  in  the 
management  of  the  program.  Discussions  and  meetings  with  Air  Force 
staff,  support  contractors,  and  prime  contractors  are  the  main  acli\  ity 
in  this  effort.  The  program  management  has  al.sc>  relied  on  e.x'ix'riencc 
with  the  jTiiis  Class  I  terminal,  designed  for  aircraft  carriers  and  other 
major  surface  combat  ships. 

The  scrond  risk  effo.rt  on  the  JTitK  was  done  for  source  selcxrtion  bc'fore 
awarding  a  contract  for  full-scale  developmcmt  in  1.981.  The  source 
selection  evaluation  Iniard  ratc'd  designated  technical  items  for  each 
bidder.  Five  color  ratings,  which  the  b-aird  defincxi  in  its  instructions, 
wcTC  to  be  used  for  each  item.  An  overall  assessment  of  tec’hnica!  jx-r- 
formance  was  rated  high,  medium,  oi  low  in  a  tc'chnical  summary’  for 
each  proiMisal,  and  the  ratings  and  the  overall  technical  summary  were 
dcK’umented. 


JTIDS  (Navy) 


The  Navy  Joint  Tactical  Information  Distribution  System  (.irilM)  is 
intended  to  provide  sc*cu re,  jam -resist ant  communication,  navigatio'i, 
and  identificainm  by  means  of  short  pulses  {isc'uaorandomly  distnii.iied 
in  time  and  frectnency.  The  terminals  for  the  system  include  a  trans¬ 
mitter,  a  data  jinn  c-ssor,  and  rexeivers.  The  terminals  are  tx’ing  devel- 
oiK’d  in  .hrtx*  c1;lss<*s — one  for  large  surface  shij)s  (such  as  aircraft 
carriers)  requiring  high-jxtwer  terminals  and  up  to  It)  voice  channels, 
another  for  early-warning  aircraft  requiring  up  to  -1  voiir  channels,  and 
a  third  for  tactical  fighter  aircraft  and  small  surfact'  shijis  requiring 
small,  lightweight  terminals  with  no  more  than  2  voice  channels. 

Work  on  the  JTIIIS  iH’gan  in  1971,  with  the  .Mr  Force  as  lead  serv  ice  In 
IfiTC),  the  program  s|i!it  into  two  phasi-s  One,  directed  by  the  Air  Force, 
is  to  de\elop  a  time  (li\'ision  multiple-access  system  (which  we  discuss  in 
tin  [)ii‘c»-ding  section).  The  other,  dinx  ted  by  the  Navy,  is  to  develop  a 
distributed  time division  inultijile-access  system  that  will  allow  simulta- 
ntxiiis  smiding  and  M’ceiving,  oiH-rable  with  the  Air  Force  s\  stem,  w  Inch 
will  not.  Fnder  the  diredionof  itte  Naval  Klectronic  .Systems  ('ommand 
in  (  rysiai  City.  Virginia,  the  .Navy  JTIDs  is  being  develojxxl  by  lluglu's 
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Atrcfiiit.  It  |i;tsM'(l  inilfsiniu"  11,  into  (iill  scalo  (ic-wloptiicni,  in  lii.Sl! 
Milestone  III  is  not  e\|HH  te(l  until  lititli. 

A  cost  an'l  sc!i.-<liil>‘  risk  analy  sis  was  iH*rfornie<1  by  a  sup(K)ii  con¬ 
tractor  in  IfiH'J,  bill  the  technical  risk  effort  has  bei'ii  handled  throu^th 
projtratn  inanat^cnient  efforts  including;,  for  example,  testing  and  review 
by  Navy  laboratories  and  mde|H‘ndent  evaluation  tJronps  in  ixni,  rc\  iew 
by  jKitential  offerors,  and  refpilar  nHH“tin>;s  of  tlu  program  office  stalf 
and  the  lontractor.  Knhanceinents  are  to  1h‘  added  through  "jireplanned 
linxitict  improvement."  Althoiijih  the  .Navy  .iriPs  pro^tram  was  not 
amoiin  the  jiilot  proj;ran.s  iisiri>;  the  Naval  Matc'rial  Command  risk  inaii- 
aneinent  system,  it  n'jMiitcdly  followed  a  similar  formal  in  a  lf'rt4  tei  h- 
iiical  re\  iew. 


Mark  XV  WF 


The  Mi.rk  XV  Identification  Friend  or  Foe  (Mark  XV  IKF)  combat  identifi¬ 
cation  system  is  intended  to  {irovide  a  reliable  mean.*;  of  ideiitifyinj;  air¬ 
borne  and  surface  targets  at  disiaiut's  eomjiatible  with  the  ran^'es  of 
"friendly"  wcajKins.  C'urrently,  the  tariiel  detection  ranne  caiiabilities 
and  maximum  ranitesof  man\  weaj sms  exceed  the  ranges  at  whn  li  tell¬ 
able  identification  is  available.  The  M,irk  xv  IKK  is  a  ciiiestion-and-answer 
system  that  will  be  •nttVKiuced  as  a  '•etrofit  to  the  .Mark  X/Xll  IFF 
.system,  the  transition  to  the  new  sys'em  to  occur  its  platforms  become 
available.  The  ,\l.irk  XV  IKK  must  \n'  ctmipatible  with  existing  systems 
because  it  will  have  to  o|K-ra’e  in  tlu‘  same  environment  as  thi  .s<.>  sys¬ 
tems  durinjt  the  transition. 

The  projtram  is  a  joint  Air  Force,  Army,  and  Navy  effort .  The  Air  Force 
is  the  lead  sei'vice  for  develojimeni.  and  manaKemenl  of  the  tiroj^ram  is 
under  tb?  Combat  Identification  System  I’rojtram  (iffice  of  the  Aeronau¬ 
tical  Systems  Division  at  \^'riHlit-i’.it'ei'son  Ai;  Force  Hase.  t)hio.  The 
system  is  in  t he  denioiisi  ration  and  \  alid.ilion  ph.ise  of  development 
The  milestone  1  review  iKcurred  m  -Inly  1  !kS  1  and  plans  call  for  a  imle- 
stone  II  review  in  fiscal  yisir  llkSS  Itoth  'lex  ts  Instruments  and  lk"idix 
CoriMiration  are  under  conlrai  t  to  i>iTfor  n  tlic  development  work  neces¬ 
sary  iH'forc-  the  systv-m  can  enti'r  ftiil  s  (e  development. 

S<'Veral  risk  effoils  have  Usm  earned  out  for  tlv  \i.irk  \v  ikk.  The  pri¬ 
mary  eiloid  wa.s  an  tesM-vsnumt  coiuliicted  by  the  Air  Force  ihii'f  st  len- 
tist  as  1.  result  of  (jiiestions  arisinit  u\  the  rex  lew  pi  in  ess  .-X  panel  was 
assemhied  to  identify  tin'  areas  of  tex  linical  risk  .and  as.si'ss  :Ih'  relalix  t' 
tis  hnieal  merits  of  aliernatixes 
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Olhcr  M.irk  W  IKK  risk  clldris  t  cnl.Tril  lui  Ihc  use  of  iiiriirm;'!  i  nj'.iin-.'i  iii^ 
jud^jim’iils  tit  (iroblfiu  ariMs  in  l  lie  systfin.  I'lic  inilrsi i me  I  rr'.  n  w  and 
tlio  di'cision  on  llif  lypt-  nf  di  \t'li)|imfnt  ftiMlracl  i<K>k  risks  inln 
considt'raiitii; 


MLIIS/TGW 


Tlio  'IVnninal  (inulant  t*  Warlicad  CRtW)  is  tine  of  tlirif  warheads  tieinU 
devel<)|M.'d  for  tlie  Mnlitple  Lanneh  [hirkel  System  ( MLKS).  'I'he  MI.KS  is 
desinni'd  tii  t'eliver  a  larjte  vtiliime  of  fire  [itiwer  in  supiMirt  of  field  artil¬ 
lery.  Tlie  T(iVV  will  enable  the  system  to  rit'strtiy  armori'd  \  i‘hiele.s  and 
tKluipment.  Ir  is  an  antonoinmis  warhead  with  terminal  hominti  and  fire- 
arid-forjtet  tarjfet  eapabilities. 

'I'he  warliead  Ml-Hs  rttw  is  a  multinational  program.  France.  (Ireat 
Britain,  \Vt‘st  tlermany.  and  the  I'nitisI  .States  are  involved  in  the 
system’s  development.  Hat  h  eoiintry  has  a  representative  in  the  pro- 
grur.t  office,  which  is  located  at  the  Army’s  .Missile  Command  in  Miints- 
ville.  Alabama.  Tlie  contractor  is  also  multinational.  Brandt  Aremimto 
( Thompson- Brandt )  of  France.  I’liorn  F.Ml  Flectronics  of  (ireaf  Britain. 
Diehl  (>  M.B.M.  of  West  (tcrmany.  and  Martin  Marietta  of  the  I  ’nited 
States  formed  MDTT  ('iin>oratiun  for  the  develojiment  of  the  varhead. 

preliminary  investigation  of  the  technoloj’y  he^tan  in  the  early  lUTtl’s. 
In  1  f>77.  the  House  .-Nnned  .'MTviees  {’ommittis*  retjuired  that  it  he  (level- 
^l|H'd  as  an  oiition  for  the  MLKS.  .-Miout  the  same  time,  the  secretary  of 
Itefense  re'tuired  that  it  be  fiurstu'd  ;ls  a  miillinaiional  program  Fol- 
lowinfi  the  sijfniiin  of  the  memoiandiim  of  understanding;  between  the 
four  countries  in  l!t7;t.  work  todefine  tlu'  r(.iW  conce()t  liettaii.  I'assiiif; 
milestone  I  in  Septemtier  the  t)ro>;ram  moved  into  w  hat  is  called 

the  comiKinent  demonstration  pha.v.  A  milestone  II  review  is  planiu'd 
for  early  lflS7. 

The  primary  risk  (‘ffort  focused  exciusi\’ely  on  technical  risk.  It  was 
made  by  the  multinational  (froiij)  as  jiart  of  its  de cussion  of  pro^tram 
oiitions  in  the  eoncepl  and  international  jirof^ram  delinilion  phase.  In 
abeiit  I  week  in  informal  discussion  based  on  the  exiH’i  ieiiee  of  iIm  mein 
tiers  and  (irior  w.irk  on  TtiW  technoloj;y  and  other  sy  stems,  the  ^trocp 
identilied  1-t  |•otenllal  risk  areas,  scri'cned  the  list,  and  ratt'd  the  risks 
hi>;h,  medium,  and  lo'.v.  I’tiis  led  to  a  smaller  list  of  •')  areas  'I  he  el fial 
was  ex<  liisivelj'  for  use  in  choosini;  the  Is'st  alternative. 

A  s<  hediile  risk  assessment  was  also  [lerformed  for  tlie  TliU’  h>  a  .sys¬ 
tems  analyst  at  the  missile  (ommand  ;ls  pan  of  the  miiltm.itioiial  elfort 
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ii\  tl\t'  (lofitiitiun  pliusi'.  Altluui^Ili  st  lifdiik'  risk  was  t  l\o  cniphasis,  i  Ik- 
analyst  said  llial  a  tts-hiural  assrssim-nt  liad  to  hr  made  licfinr  ih-- 
srlu'dr.ii'  work  coidd  1m“  done.  InlVirinatidn  front  tlii>  imiltiiiational  jtroiip 
was  folk'ftcd  and  laid  <mt  in  a  lU'twork,  Sincn  a  nnmlH'r  of  t otn  opts  were 
iH'inf;  t‘</nsid<‘rcd.  Ifii*  ;Lss4.“s,snn‘/it  was  made  a.s  f^i-ncnc  as  possjhlc  raiJirr 
than  dopondont  on  a  pariii  iilar  design  choire.  Another  analyst  atlded 
n>st  fiftiires  to  tl‘{‘  schedide  assessntenl. 


The  Arnty  Mi.\t  pro^tram  is  intendiHi  to  «‘nhan(  e  the  eaitahiiities  of  the 
Ml  Ahrains  tank.  MiAl  tleveloj'inent  la'^an  with  replaeenteiit  of  the  Ml's 
lO.o-millinteler  cannon  with  a  rjd-inillinii'ter  version.  The  effort  was 
exitanded  to  lieveloi)  armor  for  iirolectinf’  the  tank's  mohdii y  and  fire- 
po\ver  and  an  air  distribution  system  for  protect iiiK  it  again.ii  mu  iear. 
biological,  and  chemical  warfare.  Ammunition  is  to  Iv  deseloiHsl  and 
cannon  comitonen's  are  to  be  bviilt  for  the  1 2()-miriimeter  >tnn  to  ensure 
its  interchanneabilily  with  the  West  Gi'muin  lasipaid  2  tank.  I)e\  eloi>- 
ment  is  under  contract  wiIhGeneral  Dynamics. 

('nder  the  Tank  Automotive  Command  m  Warren,  Michigan,  the  Ml 
bcftan  prototN  («•  development  in  I!)?.']  and  entertni  full-si  ale  en)tinisTin>t 
develoiunent  d  years  later.  During  this  phase,  the  1 2'.)-:mllime'.er  ftiin 
was  incoriKiral<-d  into  the  developmi'nt  effoil.  The  baseline  .Ml  passed 
its  milestone  III  i»ri>duetion  de<  ision  in  l!>7tt;  tlie  MlAi,  inrludm)t  the  Hun 
and  other  enhancements,  passed  milestone  111  in  l!tS}.  Knh. iiv. fluents 
are  to  la‘  plursv  d  into  prodactioe  over  the  next  sc-vera!  years 

The  primary  mi.m  risk  effort  was  a  s«-ries  of  thris>  Tltvi;  anal\  m>s  [ht- 
fornuxl  m  I!IS2,  Hi.S.l.  and  IMHd.  Staff  memlx-rs  rejaiiiedly  con.^lderl•d 
ti'i  hnical  risk  when  they  estimated  cost  inputs  for  the  analyses  and 
came  up  w  ith  estimates  of  luHh.  low,  and  most  likely  cost  for  each  Ml  \  i 
enhancement.  TltVK  was  used  tosiipjain  ajiplicatioris  for  ri'-k  tunds  out 
not  to  Huide  technical  decisions  w  ithin  the  proHram  office  or  a',  higher 
Uwels  of  re \  lew  Two  other  activities  Huided  Us  hiucal  decisions  an.d- 
yses  of  test  results  a. id  informal  staff  discussions  ;\( cordin'..',  to  l  he  pro- 
Hram  manager,  "ad  hm  risk  assessment,  conscious  or  unconscious,"  has 
la-eu  (iai1  of  the  daily  r<putine. 
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J  la'  .X  .osT.Mt  I  M  r  t^jinnniciii  is  part  of  tie  .N.W  .S’I'.XK  (ilohal  I'osii  loniiu’ 
System  ((il’S).  a  span’  b.ised  radio  navigation  s\  stem  consisting  <il  s.iicl 
hies,  sao  ilite  (  oiilrol  and  monitor  stations,  and  e<|uipincnt  lor  their  use 
The  Cl’S  is  dcsiHiied  to  provide  v.orldw  ide  lliree-diniension.d  posii  loii 
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an<l  %  ('l()(  ity  and  iinn  jTsal  ciKii  diiialcd  tinin  jnfonnat  jon  I  hv  >./sU  ni 
can  ojH'ratc  in  all  wcadicr  and  1i;ls  a  !ii>;li  rcMstanco  to  j  •niinnf’ 

The  usi-r  equipment  <  onMslsof  a  rin  eiver  and  pr()cess<c  n  i. 
antenna  system,  a  control  display  unit,  a  flexible  hkkIh'.  "  mferfai  and 
an  optional  data  loader.  The  equipment  is  designed  to  reti  .ve  and  !>ro- 
cess  either  simiiiiaiu'ons  or  se<iiient:al  data  Irom  four  diflererii  satel¬ 
lites.  The  user  (spn|>ment  measures  veWn  ity  a/id  ran^:e  \/ith  resix-t  t  to 
each  satellite  to  den.e  the  user  s  thrts-dinu  nsional  j«)‘,ition  and 
\el(K'ity.  It  then  j>ro(  e.s.se.s  the-  data  in  terms  of  an  eattli  <  entered,  earth- 
fixed  (  (virdinate  system  and  displays  the'  information  in  jitsinraiiluc  or 
military  f^rid  cxKirdinates.  Maftiunaix  Advanced  I’rorhu  Is  and  Systems 
Company  and  the  Collins  Covernmenl  Avionics  Division  fif  IbK-ksvell 
International  are  Ixah  under  eontraet  for  the  development  of  the  umt 
('quiimient.  The  two  wdl  eom|H‘te  for  the  pnxluction  phas<-  of  the  pro¬ 
gram,  with  the  |x)ssihihiy  of  taking  a  leader  follower  approach. 

The  system  is  a  joint  effort  of  the  Air  force.  Army,  Navy,  and  Marine 
Corps,  with  the  Air  Force  fuiuliojupg  as  the  learl  service  for  develop- 
merl.  The  program  management  office  is  part  of  the  Air  force  Spa'e 
Division  in  HI  .Vgundo.  Califo.’'nia.  In  the  lull-si  ale  de\  eiojanent  i>li;ts<>, 
the  ns<'r  e<iuipmenl  was  sclmduleii  for  a  milestone  111  review  in  May 
IhHt).  The  system  pa.sM‘d  a  milestone  11  review  in  Jh7!f 

The  program  offi<  e  has  tonsidered  the  schedule  and  cost  of  Its  hnical 
problems  tts  ongoing  man;igement  of  risks  Testing  has  Iven  ernphasi/tsl, 
and  test  results,  rehahility  n.easures,  and  sulqecln  e  judgment  have  lieen 
conihinetl  in  order  to  nlentify  let  hnic.d  risks  The  jirogr.im  office  also 
isindiK  ted  a’-  exanunalion  of  tet  hnical  risks  in  at  eordani c  with  Air 
Force  Hv  ■■  lon  70-l.'>  for  the  soon  t*  selet  lii>n  for  proilueiion 


The  Army  Kemotely  I’lloietl  X'ehit  le  ( Ki’\ )  has  a  long  development  his¬ 
tory  that  Is'gan  in  l!i7-'>  Currently,  it  is  being  (levi'loix'd  by  I/n  klussl. 
Its  high  teclmologv  snhsysiems  uu  lode  forw.iial  lotiking  mfrc.cd  i.idar 
and  an  anil  .lam  tap.ibilny.  I H-v flopnient .il  work  on  the  1<I’\  Is-gaii  in 
li*7i>,  oiidci  the  diret  turn  of  itie  .-Vimv  .-Vvialion  .Syslems  Command  in  St 
Dmis.  Missouri,  but  l  be  urv  <|id  not  Us nme  a  ma|or  sys'i-m  requiring 
milcsl(/ne  review  until  DU.f  Its  firsi  nul-'sioue  w;)l  tw  Die  prodm  lion 
tits  ision,  at  nulesi'ine  111.  in  IbSii 

.■\  tits  ision  risk  anaiv  sis.  coniiin  ted  in  ptHl  ami  u|>d.ited  m  IbKJ.  l;is:i, 
and  IbSl  I  ovfiisl  schedule  iisk.  assigning  b:gb,  |o\v  .  ami  most  likely 
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csl  111). Ilfs  ot  tlllU'  ri‘»lllir«  (I  l<i  (  oni|i|i  li  (ii-M'lopIli.'Iit  1  In-  1!*S1  ;iii;ilysis 
and  ilic  1!>S_'  ii|Kl,'it<‘  idrd  ralinjis  of  icclmjca'  risk,  li.tsin;;  thnn  on  a 
(|in-sl  lonn.iin-  f>>mi>l<'I<-d  l>y  tin-  tft  ltni'.  al  staff.  'I'l  c  lii’\  su!>s\  sit-ins 
\st-ri-  rati-d  on  .i  st  all-  tliai  iiu  liidisl  (lua'  lalivt-  lalx-ls  and  t,t'tl)a'  '  rip- 
tions  of  risk  catf^lorii-s.  phis  prohaliility'  rankles  for  ll-.t-  liki-lihot 
falling  to  iins-t  (H-rforinant  i-  n-timri-nii-nts,  Tlic  scalt-  dilft-ri-U  slip 
from  yt-ar  to  \t-ar.  In  it  raapt-d  Irom  ‘'nom  or  vt-ry  low”  risk  (h-ss 

than  a  o-fH-rccnt  t  haiii  t- of  not  iiiis  liiip  rtsjiijrt-iiu-iUs )  to  "iinat  tvptahly 
hivth"  risk  I  prt-alt-r  than  a  .Mi-jx-rt  t-nt  t.haiu  i’)  In  li'Sl ,  the  scale  raii^fed 
fmin  "none  or  low”  (not  moie  than  a  llt-iH-rcenl  cltaiu-e)  to  ”iinaccepl- 
ahl>  hi>fh”  (greater  than  oo  |M-rcent.  as  in  HtKl )  Questionnaire  resnlt.s 
wt-if  .s.imined  into  a  sinph-  ratin^i  t  hiftli,  miHlerate,  or  lose )  for  i-acli 
stihsystem. 

Otiii-r  risk  it-lateil  at  tiviti<-s  for  tin-  Hr.  .  Itule  :i  decision  risk  tinalysis 
tomiilete:!  in  H'TS  ami  Tlcv  t;  analyses  in  lfi.S2  ami  Itth.’J  for  the  piodtic- 
tion  pha.se 
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The  Short -Uan^te  Air  Ikdei.-a,-  (.'omniami  and  Control  Ss  stein  (sii(>ii.MK'j) 
offt-rs  atitontali-d  coiuea.o-.d  anil  control  fnnctions  for  the  Sll<  '!iAi>  tia'a!- 
lion  Compoteis.  display  devn  es,  soltvsart-,  and  inlei  face  iHjtiipnient  are 
intendi-d  to  autom.e.e  t  In- 1 olleilion,  protessinu.  distrilniiioii,  and  display 
of  intorniation  for  Sil<  >1^X1)  weai>ons.  No  existii-.p  system  (K-rfor;Tis  these 
fun; Hot'.* ;  sonu-of  tin-nt  ‘  an  N-  jK-rforrnetl  ni.umall.c,  hut  this  is  slow  and 
tinreh.ihle  'I'ln-  pro^:ra^\  offn  i-  is  at  the  Hallisiic  Missile  iK  lense  System 
Ctimm.iml  in  liunlsvilU-,  Alahama. 


In  ,hil>  l!*Sl ,  an  ai  iiuis|-|o;i  strategy  wits  ;nipro\  tsl  li\  a  (tencralaifficer 
re\  iew  ,  w  hu  h  was  siipporlisj  hy  an  Army  in-firiKcss  r'-vii'W  in  .Xpiil 
I  l*S J.  Howev i-r,  the  <  onpr'-ss  a<  ce|>!e(J  neither  the  schedule  nor  the 
fomliii^t  reijiiirt-tm-nts  lor  the  I'.tsd  fiscal  t  ear  approjiruition.  'I'he  Ct.n- 
iti ess  did  .1;  know leilpe  tin-  mssl  for  .in  autom.itiHl  command  and  i  out rol 
system,  .iml  m  r<-s|K>iise,  the  depn(>'  iindi-rsecretary  of  I.'elense  for  tom- 
maml,  conliol,  commonn  .Uions  and  inlellipeme  apiproveil  a  restnic- 
tiirimtof  tin-  [iropr.iin  m  .Xf'iil  1!>S;1  r.ud,".et  reductions  led  to  another 
resirm  tin  init  in  the  spring’  ol  JUS  }  and  siill  another  tn  e.u  1\  summer. 


1  hn-e  ri'-k  effoits  h.ive  In-t-ii  completed  for  the  MHUiMn  riu-  first .  com- 
pil  led  in  .lanu,ir\-  l'"sl  Pv  a  systems  anal  -  at  tin-  missile  l■('nananfl, 
flit  usi-il  on  ( list  a-nl  SI  l.edule  l  tsk  .-Mthoupu  tis  hnieai  l  isk  Was  i  onsid- 
ei I'd  111  the  s<  hedille  .ind  < «ist  assessnn  lit .  onl\  ffie  <ns|  .-iitd  v  hediile 
.isjs-i  ts  were  diK  imieiited  Hie  two  other  elforts  were  made  in  ii  sjionse 
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to  tlif  iini>;r;m)'s  ii-<iriu  tMriii>;  i'lir  •.(•i  ninl  risk  cllorl  \\  .is  i  nniplrt :-(1  iii 
Au>;iisl  1  ‘iS  I  ill  lit  liM  i:s»-/l  111!  1  III'  M  Ill-Ill  I  !i‘  ri'k  nl  I  III-  SI  il  1  \v  .(I  I- 
I  Ill’ll  I  It  M  iis  I M  li  I  m:  Ill’ll  |iy  ii  siipi  Mill  1 1  ml  r.ii  I  iir  tu'  Im  t'  nw  iil  lir;i  \  ils 
friiin  I  111’  I’iiiiu-r  i  t  furl  Nu  iiu’iiimii  m.is  iii.uir  nf  llw’  It’i  hi’.iriil  risks  of 
itii’  sysii'iii  ill  till’  liiH  iinii’iilaliiin 

'I'lii’  lliinl  I’ffon.  niiiipli’lisl  in  M.iirli  lIH'i,  was  tin’  pi  iin.ii  y  risk  cffn-’. 

It  w.Ls  niiiilr  by  a  ilit  frri’iit  siip|Hii  i  l  uiitr.ti  tnr.  w  Im  iisisl  ihi’  s\  si  i  ir.s 
(‘iiliiiii'i’rinj;  in.iii.i>;i’P.ii’iil  ^iiiiiii’  piiblishcil  liy  llu’  I K’li’iisi- Sys* rins  M.'ii- 
a);;i’iiii'iit  ( 'iilli’^li’.  1‘nib.ibilitirs  nf  l.nliiii’  assi^^ni  il  In  b.irdu iin  and  soft- 
wiiri'  rnrii] Miiii’iits  wrri’  b.isi'ii  on  ilii'ir  di’iins’  nf  niiiinniy.  i  lui.’pii’Mty. 
and  di’iM-nili’iiri’  nn  inli’rl.n  in>;  ili  iiis  Tin’  prnhabilil  v  iissi^nini’n' s  •.•,,  ri 
snbji’i  tni’  but  basi’d  mi  stiiinl.inls  din  iiini’iiii’ti  in  tbc  i’ni4.i'’.’i’rmk  sHidc. 
.Stand. uds  for  bu'.li.  ini’diiiin.  .mil  Inu  wi  ri’  ;tlsn  diK  iimi  nti-d.  I'lir  snp- 
jmrt  iniilrai  tnr  inrnrix.i iilisi  ti'i  iiiiiral  risk  m  tlii’  msi  ;md  si-pi’diiln 
aii.il\  si-s 

I'lii’  Short  ■Hiinj;i’  Ait.n  k  MissiU-  II  (sirwi  n  i  is  U’lnif  di”.i’ln|Msl  tn  ri  placi’ 
till’  I  iirri’iit  .'sJinn -H.in^ti’  .-Mlark  Missilr  and  is  inlrndi’il  tn  supj.,irt  jM'iir- 
tratin>t  iKtinbi’r  missions  ilirnii;’,h  ibn  l!'l*'.i's  and  brNond  i'lii’  |h’IU’- 
I  r;d init  l*onil>»’r  iiiissimi  is  an  rsviil  i.d  l•Jl’nu■n!  of  t  lin  si ;  ,iti-i;i'  i  nail  o. 
land-,  si'.i-.  iiiid  .iir-b.isi-<)  ili’li-iisn  I  |.i-  sii-.v,  ;i  is  inii-ndi-il  to  oi n\  idi’  tl.r 
Jl- 1 1!  a.'id  adv  .nil  111  ti'i  bnnloi^y  boinln  i  s  m  i|  h  ;i  sijjm’Iso!’),  air-  n  i'.rouiiii 
niii  li-.ir  nussili-  d<-s:,',ni’d  to  alt  n  k  IimsI  .ind  ri  liM-.ii.ibli-  lan'i  is  '  hr 
sy  St  rill  (-nnsisis  nl  tin’  inissih*.  siipjim  |  i-ipiii>iii>'nt ,  mission  |  ijannm.' 
ispnpnii'iit,  and  i.iirn-r  intril.ui-s 

'I'lir  stpwi  It  Is  !s-iny  i'i-\ i-lojM  d  .d  iii'b’  I’.ilirpson  An  Forrr  H.tsr,  ( ih'.i, 
in  t  hr  Air  I'om  r  Ai  i  on, nil  a -.d  .“si  siriiis  1  is-on  'rbr  s\  sir  in  siail  u  .is 
approN  rd  in  I'.'sd  itb  .in  .u  rrlri.iir)  dri  rloi.mrnt  appro, n  !i.  no  imlr- 
stnrir  I  ir\  n-w  ii  ,is  l.i-li),  ,ind  no  disi  ri-ir  drfnonstr.ii  ion  ,i.-)il  \  .il.d.it  a  m 
pliasr  b.is  Ihs  u  i'o!.i!iii  Ii-d  h’.il  lii’i  ,  ,i  sii- v.''i  ’p,-.  I  nil  -si  ,ilr  dr\  r  lopnii-n! 

r  ffo;  t  Is  uadi  r  II  ii  's .  r.  it  h  .1  iiiili'sioni  -  1 1  1 1  i  n  -n  ;  lanm-d  lor  1 1  'Si  i. 

I'lir  prmtr.iin  olla  i-  ’..m  irln-d  In-  ii  ii\  f-'t  In  hnii  .d  risk  inlorm.ii  io,-i  on 
past  Worn-  on  •  ;!,,,i.ir  s\  ..ii-irs  It  lots  ■  oi-.ili:-,  I'-i!  .in  inlorni.il.  snbii-rt  i\  r 
,issi-ssir,i-iii  ol  .nl  s.ji.si  arms  |.i:i  mti-iiils  to  rooonsidrr  this  assrssmnit 
apl  'I  ",n  h  b  -:  b  d)  s<  .d--  dr  \  rloj  aiiriil 
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SSN-r53  submannfs.  now  nndor  construc  tion.  It  is  dcsiRncd  to  mcrf^o 
sonar,  sc'nsors,  fire  control,  and  other  control  units  int(<  an  intc^jralcd 
systt'm.  Oriftinally  a  "prt'planncd  product  improvciVicnt"  system,  with 
two  upjtradcs  of  the  hitsic  vcr.sion  foi  future  submarines,  the  si  ii.v^  has 
under>»one  a  total  restnicturinj’  l>ccause  of  problems  in  developing  a  rev¬ 
olutionary  fiber  optics  data  bus  to  connect  the  system's  computers 

The  srn.v's  projtram  is  Umiir  de\(>lo|xHl  by  the  .Saval  Sc*a  Systems  Tom- 
inand  in  Crystal  City.  Virginia.  It  was  initiated  in  Novemln-r  IMSO  with 
the  ajiproval  of  a  mission  eli'inenl  needs  statement  by  the  sc-en’tary  of 
iV'fc'nsc'  .-\t  nuli’stone  II  review  in  Sc'jitcmlKT  1083,  full-scale  develo|>- 
rnenf  was  approved  and.  in  Dect'mU'r  108.3.  the  Federal  Systems  Pivi- 
sion  of  HIM  recc'iv*  d  the  contrad  award. 

The  risk  effort  refwirted  in  our  intervic'ws  with  staff  in  the  ['ro^jr.-im 
offii-e  was  part  of  a  vieslule  risk  ass4*ssment  ix-rfomu“d  by  \av  al 
I  nderwatiT  Systems  Command  and  IBM  in  ri'sjHinv’  tc'  a  rc-quest  m  Feb- 
p.iaty  1083  from  the  assistant  sesretarv  of  the  Navy  for  a  quantitative 
aiuilysis  of  the  risk  to  ship  delivery  date's.  IBM  and  the  .Nav  y  command 
worki'd  inde|H  ndemly  to  identify  ep.tK  al  items,  including  items  oMerinjl 
a  "simufieant  tis  hnic;d  i  hallen^le.  "  and  these*  items  sc  rvi'd  as  the*  basis 
of  a  network  analv  sis  ( inipnallv  desijjn«*d  to  lx*  on^oink.  the  a,vsc*ssmeiit 
was  diseontimnsl  in  f  >isc*nitH*r  10.8.1 


I'lie  muliifacetisj  T-d"'  Trainmn  S\  stem  (T4->TS).  or  the*  Naval  Cnderkrad- 
uate  .let  Fliitht  Trair.ink  .8ysiem.  <  onsists  of  aircraft,  simulators,  a.  a- 
demie  eiiiir'a*woi  k.  and  traininit  m.ma^ement .  It  is  mtendecl  for  i  he 
intermediate  aiiil  adv.ineed  phases  ul  me  naval  flijlht  tramih):  |iii'>;ram 
for.iet  airi  raft  jalots  The  T-4.">  aireralt  is  a  twl^tandem■seal.  |e^•elvJlne 
tralner  designed  ,ind  built  in  (ireat  Britain  .A  version  w  itb  the  i  ap.ibiln  v 
of  o|H'ratin^  from  airer.ilt  <  arners  w  ill  tx*  built  in  the  I  nited  States  for 
the  .Navy  b>  .Mi  (hinneil  hoii^i.is  Corjxiration 

.\eeeleiated  development  of  the  svstem  Is  under  'lie  gind.ini  e  of  the 
Naval  .Air  Sv  ^lene  <  onim.ind  in  (  rv  sial  Cit  v  ,  A'lruinia  Combined  mile 

'.tone  i  .md  II  l  ev  lew  s  were  lii.ide  in  <  K  IoIht  1 1*8  1,  and  the  se-  I  e!  al  V  of 
1  >e tense  .H>|  •!  oV  ed  lull  .lie  dev  elopn lent  III  I  >eei*ml)er  1081  .\  ililleslolie 
HI  rev  lew  is  SI  heilllled  tor  llsi  .tl  Vear  loss 

I'he  projtr.im  olti-.  e  has  an  on^toini;  risk  inanakemenl  el  bul  F.iiiph.i 
si.'i'i^  reli.ibitit  V  t  ill*  enyiiiis  rim;  si  .||  |  has  monitored  risks  hv  nie.iiis  of 
ti'sis  o)  probli-ai  ar«*as  In  .iddit  ion  to  t  his  el  lort .  the  pi  o^i  ,tm  ol  1  u  e  w  as 
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directed  to  use  the  template  system  of  risk  m:uiaHement  detailed  in 
“Solvitij*  the  Risk  hxpiation  in  'FransitioninR  from  Development  to  Pro¬ 
duction”  (OOP  -12-lo  7-.M),  which  it  did  in  the  milestone  l-II  review. 


Trident  II  (D5) 


The  Tndont  It  (I>n) Strategic  Weapon  System  is  intendwl  to  improve  the 
performam-e  of  submarine-launched  ballistic  missiU-s  A  follow-on  to  the 
Trident  I  (C4).  the  Indent  II  fl)r»)  will  reportedly  provide  a  larger  missile 
with  greater  accuracy  and  Ix-tter  payload  It  is  to  lx*  d*'i)loye<l  on  newly 
const ructtxi  SSHN-72<)  (OHIO)  submannt's  and  backfitted  on  other  sub¬ 
marines  of  the  sjune  class  that  originally  carried  the  Trident  I  (C-1).  The 
contractors  invoivtxi  in  Ind.-m  !l  dc,)  development  include  Drafx-r  1-alxi- 
ratories  (Jenerai  K'ec’ric.  Interviaie  Kl'^etroriici;.  Lockhex’d,  S^'crr}’.  and 
Westinghouso. 

The  Navy’s  Strategic  Systems  Project  Office  in  Cn.stal  City.  Virginia. 
manag<*s  the  devc'lopmeiil .  priKluction.  and  supixirt  of  the  Tnd<  nt  ll  iir>i 
Strategic  Weai>on  .System,  which  tx'gan  full-sc  ale  development  after  the 
milestone  11  review  in  September  IPS.'!.  A  mili.^stone  III  review  i.s  sc  hed¬ 
uled  for  Marc  h  !PH7. 

The  program  management  offiee  has  used  a  nsk  management  approach 
for  addressing  tex  hnical  risk,  examining  low-ri.^k  tcxhiiologies  as  miu  h 
as  [Kissihle.  Tlu'  offiee  identifies  problen.s  through  a  stcXTing  group  that 
includes  viiior  contrai  ioi  jHTsonnel  in  ordoi  to  promote  an  exchange  of 
information  bedwex-n  the  eontraitors. 

.•\n  "improc  ed  acc  uracy  program"  wa.s  c ompleuxl  in  1P.S2  1  his  was  a 
sjHx  ial  ass*‘N>nient  of  the  tec  hnology  of  entieal  c-lemi-nts  in  order  to 
determiiw  th“  feasihihiy  of  ac  liie\  iiig  the-  ex|Hx  tc-d  imin'ov cm-  nts  in 
accuracy  of  lie'  Trident  11 

.•\  sc'par.ctc'  sc  hcxluU*  risk  assc-ssment  was  ixTlormcxl  in  lf>S:l  oy  a  su]>- 
[xirt  (ontraclor  The  ass4•^•,^lent  cmis  aiincxj  at  determining  sc  htxiiile 
risks  for  tlie  d''h\cT>  of  go\<'rnnu‘nt-fiiniish<xi  ixjuipment  ami  informa¬ 
tion  for  xulimarines  under  ccmstnietion 


V-22  Osprey 


The  \  22  <  >'-pre\  program,  formerlx  .loml  \  e(ti(;il  Lift  .Xinraft  i.D.Xi 
program,  is  an  effort  in  dexclop.  prixluce.  a.n.d  deplnc  a  mull  inussion 
\eni<  al  lake-oft  and  l.mdmg  am  r.ill  eombming  the  c.iiiabilil les  df  a  tin  - 
iHiprop  aircralt  \\  ii  h  t  le  'sn  i>l  a  lielieopler  1 1  uses  a  tilt  roi  or  t  ii.it  allow  s 
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vertical  lake-ofl  and  makes  a  transitinn  to  horizontal  f  liftht  by  means  of 
tillinfj-enftine  nacelles. 

It  is  a  joint  prot*ram  of  the  Navy.  Marine  Cotrs,  and  Air  Force  and  is  to 
fulfill  a  different  mission  ri'quirement  for  each  st>rvice.  The  Navy  is  the 
lead  se*r\  ice  and  fills  the  pnaurement  role  for  the  .Marine  Corps.  Tlie 
manap'meiu  cT  the  tiroftram  is  under  the  Naval  Air  Systems  Command 
in  Crjstal  City.  Virginia. 

An  initial  oix-rating  capability  that  would  replace  the  Marine  Corps 
mtxlium  itssault  vertical  (ift  flirt  is  planned  for  19'M.  The  program 
passesl  a  mili«Uone  I  reviiov  in  Deermbcr  1982.  with  lk‘Il  Helicopter  Tex- 
tTv/.i  ;md  IkH’ing-Vertol  under  a  joint  amtract  for  development. 

The  program  office  uses  an  ongoing,  informal  jirocess  of  risk  a.s.sossment 
carried  out  by  the  engin«s‘nng  staff.  As  tes  hnical  problcnus  arise,  tl.ey 
are  discu.vs<>d  in  routine  staff  mw.ings  h^rlier,  in  an  effort  to  determine 
the  most  feasible  ttrhnica!  apiiroach  for  the  system,  a  joint  tee  hnology 
as.sessment  group  examnu'd  risk  as  part  of  its  evaluation  of  helicopter 
versus  tilt-rotor  dt*Mgns 
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THC  UND£»t  S£CR£TARV  OF  DEFENSE 
tVASMINSTON  t>C 


9  DEC  fies 


Mr.  Frank  C.  Conahan 
Director,  National  Security  and 
International  Affairs  Division 
US  General  Accounting  Office 
Washington.  DC  20S48 

Dear  Mr.  Conahan: 

This  IS  the  Department  of  Defense  (DoO)  response  to  the  General  Accounting 
Office  (GAO)  Draft  Report  ‘Technical  Risk  Assessment  •  Unclear  Poiicy  and 
Inadeguate  Practice  Chaiartente  Current  DoO  Efforts'  dated  October  3,  1985  (GAO 
Code  973193/OSD  Ca.e  No  88S8) 

The  DoD  generally  concurs  with  the  draft  report  The  DoD.  however,  does 
not  agree  with  the  GAO  s  emphasis  on  technical  risk,  without  concomitant 
consideration  of  cost  and  schedule  risk.  The  relationship  of  all  three,  (cost. 
Khedule.  and  technical  risk)  must  be  recognited  and  balanced  in  the  management 
o*  overall  program  risk.  Specific  comments  whicfi  address  the  repoa  findings  arc 
attached 

We  appreciate  the  opportunity  to  comment  on  the  report  in  draft  form. 

SitKerely, 


Attachment 
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GAO  DRAFT  REPORT  DATED  OCTOBER  3,  1985 
(GAO  CODE  973193)  OSD  CASE  6858 

•TECHNICAL  RISK  ASSESSMENT:  UNCLEAR  POLICY  AND  INADEQUATE  PRACTICE 
CHARACTERIZE  CURRLNT  DOD  EFFORTS' 

DEPARTMENT  OF  DEFENSE  COMMENTS 


FINDINGS  I 

O  FINDING  A:  OoD  Hp«  Idpwtrfipd  Tethntcal  Ritfcf  Ai  A  Factor  Itpdinq  To 
Sch»duT^lippaq«  and  Coil  Growth.  The  GaO  o6s»fv»d  that  ipthmcjl  riVlu  «rp 
inherent  in  the  oevtiopmpnt  oi  new  weapon  tytteim  when  perlormance 
requirements  enceed  the  capabilities  o(  current  technology.  According  to  GAO,  if 
r>ot  anticipated  and  managed  m  the  eariy  phases  of  the  acquisition  process  technical 
risks  can  have  profound  effects  on  program  costs  and  schedules  GAO  described 
technical  risk  assessments  (1 RA)  as  the  process  for  identifying  and  evaluating  the 
potential  for  performance  problems,  drawing  a  distinction  between  technological 
risk,  and  program  risk,  (which  also  includes  schedule  and  .ort  risk)  GAO  reported 
that  DoD  has  identified  technical  problems  as  a  maior  factor  in  cost  growth  and 
schedule  delay  and  has  reported  that  the  level  of  technical  risk  directly  affects 
decisions  on  further  development.  The  GAO  further  reponed  that  it  is  the  DoD 
postion  TR  A  can  significantly  reduce  the  overall  cost  of  acquiring  new  weapon 
systems  The  GAO  concluded  that  substantial  Mvmgs  could  be  eipected  from  the 
cfesign  and  implementation  of  TRAs.  (pp.  i,  1-1,  p.  >7/GAO  Draft  Report) 

DoD  Comments: 

Partially  concur  The  DoD  does  not  concur  with  the  GAO  report  implication  that 
technical  risk  should  be  emphasized  in  isolation.  Technical  risk  is  only  one  element 
of  overall  program  risk,  which  also  in  Judes  funding  and  cost  risk,  as  well  as 
schedule  risk.  It  is  essential  that  consideration  of  program  nsk  include  balanced 
consideration  of  each  of  the  risk  elements.  The  DoD  does  concur  that  early 
identification  of  technical  risk,  as  well  as  other  types  of  risk,  should  reduce  program  i 

costs,  ft  will  also  improve  program  stability  and  help  ensure  on-schedule  contract  ! 

performance  Since  1981.  m  fact,  the  DoD  has  been  placing  much  greater  emphasis 
on  program  risk,  and  has  reduced  tost  growth  to  about  one  percent  ft  should  be 
noted,  however,  that  early  identification  of  risk,  while  reducing  the  amount  of  cost 
growth,  does  not  necessarily  reduce  program  tost  ft  may  simply  cause  recognition 
of  additional  cost  initially  thereby  precluding  n  from  being  included  in  grov^ 
calculations. 

o  FINDING  B:  Despite  Initiative  'it ,  t^D  Poficies  Regarding  Technical  Rid. 

Assessment  Reirsain  UmocusesTTT^  GaO  fourid  that  initiative  It  of  DoD's  1 9#  t 
Acquisition  improvement  Piogram  called  for  the  use  of  quantitative  technical  risk 
assessments  to  support  the  budget. ng  of  funds  to  cover  risk.  The  GAO  noted  that 
many  quarrtrtat've  approaches  aie  '^escribe  i  in  a  handbook  developed  by  the 
Defimse  Systems  Managerrsent  Cohegr  (DSMC)  m  i  -spometo  Inmatwe  1 1.  and  that  i 

other  quanptabve  and  qualitative  loots  Me  awaiU  The  GAO  further  noted  that  > 

there  are  many  resources  avai'ab.e  to  aid  F.'ogram  f>  ^nagement  Offices  (PVOs)  in 
performing  .echmcal  risk  asscv.roents.  but  ai:o  ohifiv’d  that  there  are  sotrse  | 

obstaclestoaclearunderstandingof  DoOeups'cutiu  is.  GAO  pointed  tn  no 
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tonvilent  definition  of  what  li  meant  by  technical  rnk  at  well  at  no  DoD-wide 
I  definition  of  the  commonly  uted  termt  high,  medium,  and  low  ntlc  While  GAO 

{  acknowledged  that  tome  ctetmitiont  entt  for  program  ntk,  tuch  at  that  developed 

I  by  the  OSMC  and  by  Air  Force  Regulation  /O- 1 S.  GAO  nonethe lett  found  no 

I  standard  oefmition  of  technical  ntk  m  DoD  The  GAO  alto  obterved  that,  wtule  rt  is 

I  true  that  the  regulations  for  documentation  of  major  system  acquisitions  include 

requirements  that  technical  risks  be  addressed,  the  degree  of  discussion  or 
identification  of  risks  is  not  set  out.  nor  is  there  any  specification  of  the  kind  of  TRAs 
to  be  used  GAO  found  that  there  is  no  official  policy  or  guidance  calling  for  the 
application  of  specific  tools  or  techmours.  nor  are  there  generic  criteria  for  TRA. 
irsdependent  of  the  approach  used  T he  GAO  concluded  that  despite  the  fact 
Initiative  1 1  was  intended  to  promote  qualification  of.  and  budgeung  for.  technical 
risks,  in  reality  it  hat  had  little  influence  o rer  the  three  Services'  prcKedures  for  TRA. 
The  GAO  further  concluded  that  there  h.we  been  no  perceptible  changes  in  Defense 
Systems  Acquisition  Revievc  Council  (DSARC)  procedures  or  operations  as  a  result  of 
Initiative  1 1 .  In  summary,  the  GAO  concluded  that  while  the  Department  of 
Defense  hat  general  policies  calling  (or  TRA.  the  policies  are  unfocused  anef  not 
clearly  deKribed  under  any  regulation  or  direaive.  (pp  2-1  through  2-17.  and  p.  S- 
2/GAO  Draft  Report) 

DoO  Comments: 


Partially  concur  The  DoD  concurs  that  policies  relative  to  discrete  treatment  of 
technical  risk  remain  unfocused,  and  that  a  generally  accepted  and  understood 
definition  of  technical  risk,  including  commonly  used  terms  (high,  rrsedium.  low 
risk),  still  does  not  e*ist  within  the  DoD  As  indicated  in  the  response  to 
Recommendation  1.  the  OoO  will  issue  a  handbook  that  incorporates  DoD 
definitions  of  risk,  including  guidance  on  what  constitutes  high,  medium  and  low 
risk,  to  the  eitent  possible  C)oD  does  not  concur,  however,  that  Initiative  1 1  has 
had  little  or  no  effect  on  the  Services'  or  DSARC  procedures  for  TRA  While  n  is 
difficult  to  identify  specific  actions  attributable  to  Initiative  1 1.  as  noted  m  the 
response  to  Finding  A,  the  overall  program  results  are  vastly  improved 

o  FINDING  C:  Differences  Among  Rroqram  offices  In  Addressing  Technical 
Ri^  From  rts  evaluation  of  JS  rr.aior^ystem  Program  managerr.ent  Offices  (?ROs) 
Tor  programs  between  Milestone  i  and  III.  GAO  found  that  no  PMOs  quantified  arnf 
budgeted  for  technical  risks,  as  called  for  by  Initiative  1 1 .  Lucking  DoD  criteria  for 
what  corsstituted  a  TRA.  the  GAO  developed  criteria  for  minimal  narsdards  of 
quality  for  TRAs.  stating  that  risk  efforts  (Rb)  should  b«: 

•  prospective.  e*amining  risks  before  proble-m;  occurred; 
planned,  not  an  incidental  part  of  program  discussions; 

-  CKpficit  in  attention  to  technical  rnki; 

•  documerrted,  so  that  the  results  of  the  assessment  could  be 
shared  with  decivcin-makers  arsd  staff,  and 

-  reocsTted  twice  in  each  acqurution  phase  to  determine  how  rtski  were 
changing 

The  GAO  reported,  that  although  all  25  PMOs  made  an  effort  to  identify  their 
technical  risk:,  only  three  conducted  risk  efforts  meeting  the  GAO-developed 
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crtttffia  ^of  TRAi  In  addition,  the  CAO  ob**efved  wide  variation  in  how  rule  eHorts 
are  implemented  Although  formatt  <o'  aisetvng  rl^k  are  generally  most  useful 
when  they  combine  a  >escription  of  technical  problems  with  qualitative  or 
cioanutive  ratinq-i  e  .  when  they  specify  the  content  and  level  of  risk.  GAO  found 
that  few  PMOs  ised  such  formats  The  GAO  also  noted  that  risk  efforts  are 
generally  most  informative  when  they  cover  all  subsystems,  not  just  selected  ones  or 
the  system  as  a  wh.sie.  yet  few  PMOs  did  so  The  GAO  also  noted  that  reliability  of 
risk  input  IS  enhanced  when  several  raters  provide  written  input  independently,  but 
again,  few  PMOs  followed  that  procedure  The  GAO  also  concluded  that  inasmuch 
as  rtsost  PMOs  did  not  consider  the  compleaity  or  maturity  of  their  systems  when 
choosing  implementation  options,  it  was  not  likely  that  their  nik  efforts,  as 
implemented,  were  as  useful  as  they  could  have  been  in  furthering  system 
development  (pp  3-1  through  3-31,  GAO  Draft  Report) 

DoD  Corrtments: 

Partially  concur.  The  OoOdoes  not  concur  that  risk  efforts  were  less  useful  than 
they  could  have  been  in  furthering  system  development  As  GAO  pointed  out  (since 
It  did  not  assess  the  actual  experience  or  degree  of  success  of  the  systems  studied),  it 
could  not  determine  whether  TRAs  anually  chosen  for  each  program  were  the  most 
appropriate  The  OoO  does  concur  that  there  should  be  criteria  for  TRA  in 
generalized  form,  with  allowances  for  tailoring  to  specific  program  circurnstances. 

It  will  not  be  possible,  however,  to  measure  all  TRA  efforts  against  uniformly 
imposed  criteria.  Each  maior  system  is  unique  m  a  number  of  respects  The  success 
of  one  system  may  be  dependant  upon  the  development  of  new  technologies,  while 
another  system  may  employ  only  proven  technologies  Thus,  the  description  nt  r:;»» 
as  high,  medium  or  low  must  be  measured  on  a  relative  scale.  r»th»i  than  on  any 
absolute  scale  TRA  is  comprised  of  a  number  oTanalyVital  tools  which  should  be 
carefully  selected  and  tailored  to  the  specific  circumstances  present  on  a  particular 
system  Prescribing  a  standard  methodology  to  be  strictly  applied  across  a  broad 
spectrum  of  individual  program  circumstances  would  be  extremely  difficult,  and  the 
desirability  of  doing  so  would  certainly  be  open  to  question.  The  selection  of  TRA 
areas  within  a  program  must  generally  be  left  to  the  Program/Pro/ea  Manager,  the 
individual  most  familiar  with  the  program  risks. 

o  FINDING  D:  Information  Provided  For  Service  And  DSARC  Review.  The  GAO 
observed  that  (in  order  to  be  useful)  deciuons  regarding  the  pace  and  direction  of 
these  programs  must  be  made  during  milestone  reviews  at  the  Service  and  at  OSARC 
levels.  The  GAO  found,  however,  that,  on  the  average,  technical  risk  information 
was  presented  in  only  80  fvercent  of  the  vanous  drscuments  for  Milestone  I  arid  in  76 
percent  for  Milestone  II.  The  GAO  further  found  that  the  analytical  approach  used 
and  the  scope  of  risk  was  almost  never  reported  in  these  documents.  After 
reviewing  briefing  charts,  minutes  arid  scripts  used  in  these  reviews,  GAO  concluded 
that  rt  was  unlikely  that  much  information  was  conveyed  orally  to  revievvers  on  the 
approach  and  scope  of  the  rrsk  effort.  The  GAO  also  corvcluded  that  milestone 
decision  documents  rarely  combined  narrative  information  wrth  qualitative  or 
quantative  ratings  for  all  subsystems  (pp  3-1  through  3-31,  pp.  5-3.  through  5- 
7/C-AO  Draft  Report) 

DoD  COTTriventsr 

Partially  concur.  The^ToD  does  not  concur  with  GAO's  methodology  for  arriving  at 
83  and  76  percent,  respectively  The  DoD  guidance  does  not  require  a  risk 
assessment several  of  the  documents  which  the  GaO  used  m  calculating  these 
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tveugn  Th»  DoD  doM  tcncur  that  where  rnk  data  it  preiented,  it  thould  be 
complete  and  the  definitioni  and  methodology  understood  by  the  reviewers.  The 
OoO  handbook  (tee  response  to  Recommendation  l )  will  provide  reasonable 
dr  hrtitions  and  methodology  as  a  basis  lor  such  understanding  The  Handbook, 
however,  will  not  legui'e  the  preparation  of  a  risk  assessment  for  every  prograir. 
Defense  Acquisition  improvement  Program  (DAIP)  imative  1 1  indicates  that  risk 
management  techniques  should  be  used  where  appropriate. 

o  FINOlNQ  t:  Difficulties  In  ConsrrsunicabnQ  And  AssetsinQ  Technical  Risk 
Information,  .n  their  discu»sions  with  programs  manaoerstPMM  and  with  otKer 
personnel  within  the  PkdOs.  the  GAO  found  that  most,  but  not  all,  PMs  were  aware 
of  the  characteristics  of  their  risk  efforts  GAO  defined  these  characteristia  as  ( 1) 
format-whether  risks  were  rated  m  qualitative,  quantitative,  or  narrative  terms.  (2) 
scope-whether  the  focus  was  on  the  system  as  a  whole  or  on  subsysterm;  (3) 
procedure-whether  input  was  ootamed  fiom  a  single  individual  or  group;  (4) 
sources  of  mput-whether  the  contractor,  laboratory,  PMO  or  other  sources  were 
relied  on  for  technical  risk  information,  (S)  approach-whethcr  quanbtative  or 
qualitative  approaches  were  used  to  determine  risk.  The  GAO  concluded,  however, 
that  PMO  staff  turnover,  and  the  failure  of  some  PMO  staff  to  mention  risk  efforts, 
even  when  they  were  cicKumented.  suggested  problems  with  information  flow.  The 
GAO  also  concluded  that  neither  program  documentation  nor  briefings  were 
adequate  for  informing  PMO  staff  or  reviewers  about  technical  risks  GAO  noted 
that  in  some  briefings  and  documents,  technical  nsk  was  not  even  addressed.  GAO 
observed  that  m  others,  risks  were  treated  mimmalit,  as  when  the  system  was  oiven 
a  qualitative  risk  rating  with  no  eiplanabon  In  addition,  further  complicating  the 
reviewers'  task,  the  GAO  found  that  risk  for  specific  programs  was  addressed 
differently  across  documents  dealing  with  the  same  programs-i  e.,  rating  scales 
changed,  and  ratmqs  themselves  changed,  all  without  enplanation.  The  GAO 
further  concluded  that  oecision-makers  within  the  program  cff ice  and  at  review 
levels  cannot  base  decisions  on  the  true  technical  risks  of  a  system  if  they  do  not 
know  about  an  assessment  or  there  is  not  enough  information  presented  for  them 
to  evaluate  or  understand  it.  and  that  ultimately,  the  risk  efforts  will  not  be 
effective  if  decivon-makersdo  not  make  use  of  iheir  findings,  (pp.  4-1  through  4- 
24) 

DoP  Comments: 

The  DoD  concurs  As  noted  in  the  response  to  Finding  C.  however,  uniformly 
imposed  criteria  across  all  DoO  major  programs  will  not  be  possible.  At  the  present 
state  of  the  art.  TRA  is  an  ertrensely  compici  subject.  Mevertheless.  witnin  a 
program,  the  definition  of  terms  should  be  uniform,  a  •.'ommon  understanding  of 
these  definitions  should  enst.  and  the  application  of  terms  and  definitions  should 
be  clear  to  reviewing  authotrties.  The  handbook  described  in  the  response  to 
Recommendation  1  should  achieve  these  results. 

o  FfNDlNG  F:  Contractor  Rrsk  tfforts.  Ir.  a  related  subjccC  the  GAO  found  that 
PMOs  orten  reued  on  contraaors  to  identify  technical  risks,  but  generally  received 
inadequate  information  on  the  contraaor’s  risk  efforts  For  enample,  GAO 
reported  that  frequently  PMOs  received  only  the  contracton'  risk  rabngs  and  did 
not  know  how  the  risk  efforts  had  been  conducted  or  how  the  ratings  were  drfm  wf. 
The  GAO  aho  found  that  PMO  naff  believed  that  contractor  efforts  may  be  biased 
because  mdunry  does  not  want  enimates  of  enreme  risk  to  jeopardiae  winning  and 
maintaining  contracts  (GAO  observed  that  this  vime  bias  may  affect  ertimatesof 
risk  within  the  program  office  or  OoD,  vnce  Defense  officials  reportedly  prefer  to 
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fond  lyrtan  ■■  with  low  to  modfrala  »«hn.cal  rnk  )  Tf,#  GAO  concluded  that  PMOt 
did  not  rec'tve  tufficiant  mfo'mation.  lo  mott  ir.itancrt,  to  evaluata  tha  aoaquary 
or  accurac-/  c‘  tha  contranort'  risk  eftorti  (p  ni.A-J,  through  4-31,  S  S,  and  5-6, 
GAO  Draft  Report) 

PoD  Comma  nu: 

Tha  OoO 't  ^turt.  In  thoM  cavn  whar*  TWA  n  condurtad  b/  tha  contractor,  tha 
informati  nn  n^asantad  thould  ba  uifficiant  for  a  compiata  uisdarnarsding  of 
rrtathodo  cp  jr,  dafmrtjon  of  tarrm,  att.  irsad  by  tha  conpacior  in  tha  anaiyiit 

O  PiwtyWG  G:  Staffino  Arsd  Training  for  Propram  Qffica  Xitk  tffo^.  Tha  GAO 
raportad  that,  m  tba  majority  of  P^Os,  ma  ttafh  wara  'in»oi»ad  in  both  talacting 
and  parforming  tha  risk  aftortt  Daspita  this,  howavar,  GAO  fouryj  that  tachmcai 
risk  attatsmant  racaivas  lirtla  attantion  m  tha  5arv«at'  training  courvai.  Tha  GAO 
fourtd  that  tha  Army  it  tha  only  Sarvica  with  a  courta  on  rnk  at  part  of  itt  ragular 
courta  offaringt  Tha  GAO  furthar  found  that  whan  nik  it  mantionad.  it  it  broadly 
defined  at  ‘program  ntk,'  and  tachmcai  ritk  it  addrattad  only  minimally  and  that 
fvaithar  tha  Sarvica  tchoolt.  nor  tha  05MC.  ditcuti  a  o proachat  for  attau<ng  tachmcai 
ntk  Tha  GAO  concludad  that,  ganarally,  ihara  appaart  to  ba  intoHioant  trair.ing 
availabla  to  support  tha  oarforrrsanca  of  TRAt.  (p.  m,  pp  4-24  through  4-28,  p  4- j  t, 
p.  5-S/GAO  Draft  Raport) 

DoO  Comrrtantt: 

Partially  concur  Tha  OoO  doat  not  agraa  that  tha  information  m  tha  GAO  raport  or 
othar  data  available  to  tha  OoD,  tupoortt  a  concluuon  that  thara  it  miufliciant 
training  to  tuDport  tha  parformanca  Of  TRAt  for  instance,  tha  DSMC  provdat  tha 
following  covaraga  of  risk 

a.  Program  Managannant  Courta  85-2; 

Inttfuctional  Unit  T2.1 130-H40  Ritk  Managamant 

b.  Program  Managamant  Courta  86-1; 

Inttructional  Unit  T2  1130-1140  Ritk  Managamant  Naw  Unit.  Ritk 
Workihop  •  a  til  hour  workthop  utilizing  parional  computer  tizad  modal 
to  evaluate  changes  end  provda  tha  itudant  vmth  an  undarttanding  of 
risk  managamant 

Naw  Unit.  Quantifativa  Mathodt  for  Program  Planning  and  0,ntrol  •  a 
problem  oriented  unit  to  illuttrata  how  the  PM  can  integrate 
parformanca.  tchuaduta,  cost,  risk  8  uncerumty. 

c.  Program  Managers  Workthop 

A  Risk  Managamant  Workshop  hat  baan  a  regular  part  of  this  courta 
tinea  iti  inception  in  ianuary  1984 

d.  1983  Dafansa  Ritk  and  Uncertainty  Workthop 
USA  5pontorad/D5MC  Hotted;  13-15  July  1983 

a.  Rftk  Attattmant  Tachmguat  -  A  Haridbook  for  Program  Managamant 
Partonnai:  fint  f  dition;  July  f983 
Oavalopad  and  published  by  D5MC. 

f.  System  Inginaaring  Managamant  Guda:  first  tdrtion,  October  1983 
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(Second  Edilion  <urr«ntly  underway)  Developed  and  pubitthed  by 
OSMC  Chapter  27  It  entitled  Ri'-k  Analytii  arvd  Management 

g.  Computer  Models  are  currently  m  use  or  m  development  by  DSMC  as 
follows  * 

1.  DPESO  Model 

2  trace  Model 

3  CASAMsdel 

All  of  the  abow  models  are  being  si/ed  to  run  on  the  DSMC  Personal 
Computers 

The  OoD  concurs  that  ‘echmcal  risk  ts  taught  in  the  conteit  of  program  risk.  The 
OoD  agrees  also  that  technical  risk  assessment,  in  the  content  of  overall  program 
risk,  will  be  Qiven  increased  emphasis  The  DoD  handbook  being  developed  (see 
response  to  Recommendation  1)  will  help  m  this  effort. 

O  FIWOINGH:  Petinitioais  Of  Risk  and  Risk  Ratmq  Cateoones.  TheGAOfound 
that  fewT'hTOs  know  hov/  the  Services  or  DoD  ootoments  define  risk.  For  example. 
GAO  reported  that  no  PMO  cited  the  DSMC  definition,  and  only  or'e  An  Force  PMO 
was  aware  of  the  definition  of  risk  in  Airforce  Regulation  70- IS  The  GAO  found 
that  while  rrvany  PMOs  had  a  definition  of  rise  shared  by  most  staff  members,  the 
definitions  varied  widely  across  PMOs  GAO  reported  trial  many  PMOs  e  ipressed 
risk  in  gualitative  ratings-high.  moderate,  low.  or  red.  yellow,  green.  Their  ratings 
were,  in  turn,  denned  m  narrative  or  quantitative  terms  For  example,  high  risk  was 
sometimes  defined  narrat/veiy  as.  'beyond  the  stale  of  the  art'  or  defined 
quantitatively,  as  at  least  an  80  percent  chance  of  failure  The  GAO  concluded  that 
narrative  as  well  as  quantitative  definitions  were  widely  divergent  across  all  PMOs 
and  were  often  contradictory  The  CaO  further  concluded  that  with  definitions  and 
ratings  so  inconsistent,  confusion  is  almost  inevitable  in  addition,  the  GAO 
concluded  that  the  results  of  any  risk  effort  performed  without  regard  for  such 
inconsistencies  wili  not  be  very  valuable  artd  may  mislead  dectvonmakers  Finally, 
the  GAO  concluded  that  the  PMOs’  current  approach  to  addressing  technical  risk 
offers  no  guarantee  that  reciuistte  rnfotmatjon  wi'l  be  provided  to  drevon-makers 
inside  the  PMO  or  at  the  higher  levels  of  review,  (pp.  4-1,  p  4-18,  p  4-31,  pp.  >- 
SrGAO  [>rafl  Report) 

DoD  Comments: 

Partially  concur  The  OoO  concurs  that  definitions  of  risk  shoukf  be  tomistent,  and 
that  these  definitions  should  be  ursderstood  by  program  personnel  as  well  as  by  the 
decivon  makers  reviewing  the  program  As  indicated  above,  a  DoD  handbook  will 
•ddres.  these  The  DoO  does  not  concur,  however,  with  the  implication  that 
standard  definitions  will  set  out  a  tmrformly  applicable  categonietion  of  nsk,  which 
will  necessarily  provide  comparab'Irty  horn  program  to  progiam  Cotnidenng  the 
diversfty  of  nsks  eocoorrteTed  ac'oss  a  broad  spectrum  of  programs,  eacfi  with 
unique  problems,  selection  of  risk  assessment  looh.  methodology  and  risk 
definitions  should  be  tailored  to  best  sort  the  individual  progiam  Once  defined, 
they  must  be  uniformly  understood  Risk  assessment  should  not  be  considered  as  an 
exan  science,  but  should  be  recognued  as  the  art  that  n  is  at  the  present  time 
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III 

Ail^KlK'r  I  fn‘m  Ih^  I 

[kr|i«niiw*flt  Ilf  IVirn*»* 


RfCOMM(KDATIONSTOTHE  CONGRESS 

O  RECQMN  (  NDATIQN  1  GAO '»COmm*ndKj  th*t  Cong'm  (Jifrtt  tN» 
p  77  S«<'ftary  orLicfniv#  lo  Of fToip  t»<l'nic*l  *n0  »ii« CJlrgo»‘H  IP  b  fc'GAO 

Df*tt  Report) 

I  DoO  Commenti 

PertiRlIy  toncu'  The  C>o0  conturt  th**  o-og'em  ruk.  to  .ntlude  tott.  vtheOulf.  »ryd 
I  te<hni<*l  fitk  together  irrith  rivk  reting  ceiego'iet.  thoold  be  dehried  The 

defih'tiont.  however,  mutt  be  MJ'r>ev»h*t  broeO.  irnce  ttrict  tt*r>o*rO  Oefimtiom  of 
fitk  rating  (at(.>go'iet  could  not  be  imposed  acrou  all  OoD  program^ 

I 

i  O  RECQMMENO^  TlQN  ?  GAO  recommended  that  Congrett  direct  the 

i  Secretary  orCleferce  :o  require  that  'iik  eftortt  frycul  eipliCiHy  on  technical  rnk,  and 

be  protpective,  planned,  and  repeated  early  and  late  m  each  acguitition  phate  (p 
on  p  'T  S-7/GAO  Draft  Report) 

DoO  Commentt: 

Partially  concur  Rivk  atvetcment  effortt  thouid  include  and  emphavire.  but  not  be 
l,-r-.it»o  to  technical  ink  lo  forui  only  on  technical  rnk  to  the  eidution  ol  cott  and 
tcheduie  rnk  deniet  the  tuong  coupling  tnat  etnii  between  triem  further, 
PioiedVrogram  Deciuon  Authority  thou‘d  decide  the  frequency  of  rnk  ptmcmenu 

'  o  RECOMMENDATION  I  GAO  recommended  that  Congrett  d'fect  the 

'  Secretary  oTCrefenye  to  require  program  cffnet  to  oocument  their  ncn  eHortt  (pp 

(onp  77  S  T/GaO  Draft  Report) 

DoO  Commentt 

I  Partially  concur  In  thote  cicet  where  a  program  perform*  rnk  acvencment,  Ihone 

ettorxc  thould  be  dcxumynted  The  OoD  doet  not  concur  that  there  ihould  tit  more 
'  documentat'on  reouirementt  on  proqrami  than  thove  ulready  detcribed  in  £>oD 

I  inttroction  SOOO  2  'Ma|Or  Syttem  Acquiution  PrcHeduret  ’ 

RCCOMMCNOATlONSTOTMt  OtPARTMEMT  Of  DEFENSE 

I  o  PE  COMMENDATION  t  GAO  recommended  that  the  Secretary  of  Delenv# 

j  ettabiith" g'uKjelme*  for  t’le  implementation  of  rnk  rftortJ,  regarding  optiOrn  lo' 

rr'f  om-mnoa  I  format,  uope,  input  p'ocedu'k .  and  atkeun^ent  approachfi  ipp  S  /.GAO[>aft 

lonku".  p  7  7  I  Report) 

( 

:  DoP  Commentk 

Partially  concur  The  DoO  concur*  that  ritk  acvrcvment  effortt  khould  be  devr. bed. 
i  but  not  in  tpecific  detail  The  D>7AC  will  be  requetteO  to  prepaie  a  DoO  handbook 

on  lire  management  of  O'ogram  rn  OetA'h  of  the  rnk  atceccment  eHortt  fO'  each 
I  program  then  will  be  determined  by  the  Service*  and  the  individual  programt. 

e  RECONfVf  NDATION  ?  GAO  recommerrded  that  .h*  Secretary  of  Defente 
reguire  that  tecmicai  rn*  inrormaiion  provided  for  review  incluoe  a  detcr-ption  of 
fovr  'C'  i.immorfla  farmat.  tcope.  input  procedure*,  tource*  of  ntk  informet’Cr.r,  and  a*ie*trnent 

ign  S  lO  (1  ,’7  '  Appr  yachet  <p  b  7/GAO  Draft  Report) 
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OoD  Cofnm^fiQ 

Nonconcur  It  ii  not  n*(«t#ry  lo'  th*  ot  Detrni*  to  rrqti  f»  »»plicit 

element!  ot  mk  f*»i»w  mtormjtion  «no  «4i«^in'*nt  (pproKlkirt  1  r<hnic*l  ink  o*  » 
ip^citiC  lyvikni  mjy  d«  v**II  undent oo<J  b*  OSARC  memben,  or  .'’•y  reou're  ekryiog 
deorett  of  exbo'elion  m  ooturrenrv  presented  purmg  revirwt  DoO  Iriitroct  on 
SOOO  i!  providet  for  tfie  iub'T'itt*l  of  det«  et  determined  by  tbe  Oefenve  Atqoiution 
tiecutive.  end  te<joeyti  for  dete  reyuit  from  •  review  o'  p-ogrem  d'Hurr.enielron  by 
veriout  tteff  elerrients  of  the  Otfiie  of  the  Serreury  of  Defenye  prior  to  DSAHC 
mnnlonei  Thiydet*.  tpoelher  with  the  gurdenre  m  tJ.e  new  DoO  hendtxrok, 
thou  Id  provide  the  betit  tor  full  urtderttending  of  other  ritk  eferrventi  invoteed  tn 
indivrduel  p'ogrem  rc-nt  wt  | 

o  COIkihf  t^D*TiON  3  CaO  refommervded  fhet  the  Serrelery  of  Defeme  ! 

thoolJ  contijei  prov.J^rig  eaditiortet.  n'ore  fotuved  treimng  m  lethn-cel  ritk  i 

•ttetvmenu  to  topport  •  greeter  empheut  on  techmcii  ritki  (pp  S-&rGAODrett  | 

Report)  I 

DoP  Corrvnentt.  j 

PertieHy  tonrur  The  DoD  <f  ntort  thet  ritk  ihoutd  be  emphep/ed  Greeter  } 

emphepl  on  ritk  ettetiment  trchmooet  rt  ongoing  in  the  foi-iert  of  overell  p/ogrem 

ritk  However,  empr-evt  thouU  not  be  fokuved  jott  on  tethf  ',ei  rrtk  ettettmena. 

which  It  only  e  pen  of  p'ogiem  nvk  The  ymphetit  rn^tt  be  e  heirr-ced  epproecn  to  i 

the  menegement  of  progrern  ritk  (Alto  tee  retporite  to  finding  G  ) 

°  RtCOMMfNQATiON  4  CAQ  recommended  thet  the  Secretery  of  C>e*ente 
require^cbnliecio'  riu  effortt  to  be  vu't'Cieniiy  documented  to  el.ow  independent 
eveioetion  end  ute  in  the  piogrem  office  (pp  S  6- GAO  Drett  Report) 

DoO  Cornmenu:  | 

Concur  Where  contrectori  pe-'orm  the  ritk  enelyvt.  th;t  eHort  vfiould  be 

tufficiently  documented  Concurrence  doet  not  imply,  however,  thet  ell  progren* 

mutt  heve  contrectori  perform  rick  ettettment  et  e  pert  of  their  confect  When 

required  end  included  m  toe  contrecf.  the  progi em  mutt  provide  eppropriete  | 

contrectuel  lengvege  end  ci'rection  to  guKje  contrector  i  eflortl  end  mvurt  e  ! 

tet  ifectory  product  Specifu  guidente  in  thit  eree  eito  will  be  covered  in  th*  i>ev» 

DoO  hendbook. 
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Appriidii  III 

Advance  (  ommentik  fn>ni  the  I  .K. 
Deparunent  of  IVfctw* 


"Finding  A"  As  requestt'd  by  th<“S<‘iKite  Ci<>\(‘mini’ntal  Affairs  Commitlff,  tiic  sub- 

jtvl  of  this  report  is  tiTlinital  risk.  This  dfx-s  not  mean  liial  w»'  sui'port 
an  appmaeli  to  projtram  management  tli.it  de-einph;i.si/A‘s  cost  or 
schedule  risk.  On  thi*  eontran.',  it  is  privisely  the  relationshi])  Ix'tvvivn 
tc'chnical  risk,  on  the  one  hand,  and  cost  and  .schedule  i)roblem_;,  on  the 
other,  that  promptexJ  c.ao's  review.  If  one  con.side."»  tec  hnical  risk  as  the 
indeiH'iident  variable  and  cost  and  schedule  problems  a.s  deix-ndent  vari¬ 
ables — on  this  jxiint.  we  af’rt'e  witli  iKip — then  the  pnqxT  treatment  of 
ti'chnical  risk  eovers  cost  and  schcxhile  risk  to  some  degree.  .Neverthe¬ 
less,  in  this  rejiort  wv*  have  explicitly  considered  cost  and  schedule  risks. 
Our  review  of  the  ph;tsc>s  of  acquisition  in  ( hapter  1  refers  to  a  w  ide 
range  f  'onsiderations  afflicting  development — lost  and  schtidule  prob¬ 
lems  a;  ell  i.s  technical  ones.  In  cliapter  2.  we  noted  that  cost,  schi'dule, 
and  tech.iu  .11  ,‘roblems  are  interdeiK'iidenl.  Thus,  we  have  focuvd  on 
technical  risk  but  hace  not  isolated  it  from  other  mqiortant  risk  eli'- 
rnents  In  resiMmse  to  ikip’s  comment,  we  revisiid  thi-v*  chapters  in  order 
to  emphasi/.e  the  imjxirtar.ce  of  cost  and  schedule  problems  ;vs  well  as 
technical  ones. 

However,  we  found  some  ixip  a.sses.sm'>pls  that  tiv'irsered  cost  or 
schedule  risk  without  differentiating  the  sourix's  of  thev*  risks  ;ls  t“<  h- 
nical  or  other  kinds  of  })roble:ns  .\]ons)\  er,  risk  assi^ssors  are  somei  tines 
asked  to  estimate  the  hkelihiMid  of  i-ncouiUenng  technical  problems 
given  siHs  ific  cost  or  m  hedule  constraints;  hence,  we  hare  emp!,asi/ed 
that  assessments  should  ex|»ressly  identif.i’  technical  as  well  ;is  cost  and 
s(  hedule  problems.  What  is  dearly  needed  is  a  balance  of  attention  to 
technical,  cost,  and  schedule  risks 

txip  stall’s  that  early  risk  assi-ssments  ran  nxluci’  cost  grow  th  but  do  not 
iKs  essarily  reduce  program  cost,  .Actually.  i  ..her  outcome  is  Ixaieficial. 
Hut  reduced  program  cost  is  certainl\'  more  likely  if  risks  are  identified 
and  monitored  carefully  from  the  outset. 
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App<*IKiiX  III 

Advance  ('-omincnU  from  Ihr  I  Ji. 
Deparunent  of  lU-irniw 


“Finding  B"  cUims  that  Initiative  1  i  reduced  cost  jtrowlh,  but  as  we  rcfK)rtcd  in 

cliapters  2  and  3.  none  of  the  2")  program  management  cffavs  has  jaT- 
formed  a  quantitative  technical  risk  assessment  for  u-  i  budgeting, 
and  Initiative  1 1  has  not  stimulated  new  policy  in  the  jr\  ices  or  the 
nSARC  regarding  the  identification  and  quantification  of  technical  risks. 
DOD  has  not  disputed  the.sc>  findings.  It  is  reasonable  to  c  -include  that  the 
reportcxl  r“duction  in  cost  growth  stems  from  other  factors,  such  as  the 
reducc'd  inflation  rat“s  in  rivent  years. 


“Finding  C”  Neither  the  current  nor  the  eventual  success  of  these  programs  is  the 

only  basis  for -determining  whether  risk  assessment  prex  edures  arc  ade¬ 
quate.  Principles  derived  from  experience  also  offer  valuable  guidance. 
Prt*sumably  .vorking  from  previous  experience,  IX)U  approvt-s  of  assess¬ 
ment  criteria  "in  generalized  form"  but  would  rc'scrve  decisions  on  "ima- 
lytical  tools”  for  agram  managers.  These  cximnients  arc  consistent 
with  the  conclusions  we  drc'w  from  our  analysis  of  program  manage¬ 
ment  exix*rience  inside  :ind  outside  ixip.  The  five  criteria  we  have  set 
out  are  generic — that  is,  they  are  aj^iiropriate  for  all  nuyor  programs. 
With  respc'ft  to  analytical  approaches  and  implementation,  we  have  rec¬ 
ognized  the  nml  for  flexibility,  since  d. 'visions  in  analysis  and  imple¬ 
mentation  deixmd  on  particular  char.uteri.stics  of  systems  such  as  their 
maturity  and  eomplexity.  Few  of  the  program  office's  have  ixTlormed 
technical  risk  assc'ssments  mcvling  the  five  criteria,  and  few  considered 
the  maturity  and  eomplexity  of  their  systems  when  conducting  their 
assessments.  Tfuis.  dc'spite  the  absence  of  data  on  the  effevts  of  risk 
assi'ssments.  it  remains  unlikely  that  most  of  the  ass<*ssments  we  found 
were  as  useful  to  program  managers  and  reviewers  as  they  might  ha\  e 
bex^n. 

We  have  recommended  that  rating  categoric's  such  as  high,  medium,  and 
low  be-  dcfiii  vi.  but  we  thought  Itiat  the  development  of  definilions 
should  be  left  to  ixu>.  pod’s  forthcoming  bandbcnik  will  apiiarcntly  pro¬ 
vide  definitions  of  technical  risk  and  of  risk  rating  catc*gorics  (sc-c  ixip's 
comments  under  “Finding  B")  llowevc’’,  we  reiterate  our  statcmc-nts  in 
chapter  4  on  the  lux'd  for  definitions  that  c-an  ^cNtppjicd  amiss  [iro- 
grams  in  order  to  ri-duce  existing  disparities  in  basic  risk  concepts  and, 
thereby,  facilitate  management  and  review.  The  task  is  to  find  as  much 
common  conceptual  ground  as  [xissible  Ix'twcc'n  programs  or  meaningful 
subsets  (for  exanqile.  iiigb-teclinology  jir  .grams)  Wc  have  reixiried  that 
the  Air  Force  tiKik  this  approach  in  defining  program  risk  in  Air  Force 
Regulation  70-15.  It  is  diffic  ult  to  imagine  a  handbook  for  general  use 
that  would  not  attempt  to  find  this  common  ground. 
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“Finding  D"  {KiiiUi'd  <nit  m  chapters  2  and  3  that  ik)1)  ref.viliitions  require 

cnvera^te  of  te«  hnieal  risk  in  tlu*  nuleslonc  diKaimenis  ixiu  Instruction 
r>0()0.2  and  tin-  Army’s  Materiel  /Acquisition  HandlxKik  (DAIiCO.M-l’  7(i- 
2)  c;dl  for  information  on  risk  areas  and  risk  rt'diietion  efforts  in  the 
system  concept  paper,  decision  coordinating  naiH-r.  integrated  p.  ogrum 
summary,  and  acquisition  strategy.  IkH)  In.'tniction  odOtl.d  rt-quin  dis¬ 
cussion  of  cnti'-al  'ssiics  in  the  test  :md  I'valuation  rnastci'  plan,  sjH-cifi- 
cally  including  issui's  arising  from  technical  risk.  This  covei  iige  is  not 
{Kissihle  \\  ithoiit  some  sort  of  risk  a.ssessment.  howe\er  formal  or 
informal,  extensive  or  brief.  We  searchi  d  for  technical  risk  information 
in  thes*‘  diKuments.  uhether  or  not  it  was  (>.vplieitly  linked  to  any 
as.sc'ssmi'nt .  but  found  no  sin  h  information  in  20  to  21  ixTcent  of  the 
documents.  It  is  imiHirlant  that  technical  risk  information  always  U- 
available  in  program  reviews  and  that  this  information  cover  both  the 
metluKls  and  the  results  of  an  as.sessment.  Ifeviewers  can  then  evaluate 
the  informtuion  and  weigh  it,  as  they  chiKise.  along  with  other  factors 
considered  at  each  milestone. 


“Finding  E"  recognize  that  technical  risk  assessment  is  complex  and  niii.st  take 

particular  program  circumstances  into  account.  lUit  nuKlern,  ‘'high  tech" 
\\<‘a(M)n  systems  are  also  eomiilex,  and  there  is  r<’;i!l\'  no  alli'rnative  to  a 
careful,  thorough  a.ssessment  of  the  technical  risks  involved  in  devel¬ 
oping  tlscM*  systems.  As  we  have  noted  in  our  res|K)ns<'  to  ikiP’s  conment 
uniier  "Finding  generic  criteria  for  risk  a.ssessment  can  tx'  develoiH-d 
and  apjilK'd  without  sacrificing  nexibility.  We  ha\  e  further  noted  that  it 
is  imfsirtant  that  definitions  of  technical  risk  and  of  rating  categories,  in 
tin*  manner  of  .-\ir  Force  Uegulation  70-l.n,  lx*  general  enough  lor  use 
across  programs  We  Iroix*  that  the  forthcoming  ikid  handlMKik  w  ill 
accommixlale  ihesi-  purjxisi's. 


“Finding  F“ 


No  resiMinsi-  is  necessary. 


“Finding  G"  1 1"'  conehisi..ns  cited  under  "Finding  (I"  are  b.i,-x‘vl  on  an  anah  sis  of  the 

courses,  workshops,  and  handhiKiks  ixip  refers  to.  The  courses  and 
work.sho[;s  eitlier  cover  tectinical  risk  minimally  oi' do  not  mention  it  at 
all.  Other  resources  describe  assessment  lei  hniques  but  provide  no  guid¬ 
ance  for  sehsiing  techniejues  that  are  suitable  for  particular  programs. 

It  is  im|K)itant  to  note  that  most  of  the  courses  ;ind  workshops  cited  in 
iXJD's  comments  rover  risk  management,  not  risk  icssi-ssment,  .As  we 
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ha\o  aotcti  in  » hapior  2,  risk  manaHoment  deals  with  problems  as  tliey 
arise,  wherejes  risk  assessment  idei  tifies  problems  in  advanec.  Thus, 
risk  assessment  alerts  program  staff  members  to  problems  that  they  try 
to  avoid  or  solve  through  subsequent  risk  management,  ixjo's  eoursc-s  do 
not  provide  enough  eoverage  of  the  coneepts  and  analytical  tools  that 
are  use<l  in  the  assessment  of  six'eifieally  technical  risks.  Several  of  our 
other  findings,  such  as  the  ineonsisteney  in  definitions  of  risk  and  risk 
rating  categories  and  the  lack  of  explicit  attention  to  techn'cal  iisk.  are 
further  evidence  of  the  inadc-qiiacy  of  current  training.  In  short,  the 
technical  content  of  tKip's  training  for  risk  as.sessment  lags  behind  the 
tc‘chiii<  ai  content  of  th?  weafxm  systems  bc’ing  de\ clopc'd. 


‘‘Finding  H”  ^'hs  mis  onstmed  our  conclusions  under  ‘Finding  H.”  We  have  not 

ca!le<l  I'or  a  standard  set  of  concepts  and  tcKiis  to  he  applicable  in  all 
respects  to  every  program.  We  have  noted  in  chapter  3  that  technical 
risk  a.ssessment  rcquirc's  some  dec  isions  I  hat  cannot  be  considered 
generic.  Various  concepts  and  tcxils  in  technical  risk  a.ssc'ssmcnt  arc, 
moreover,  not  iiiii.'-irmly  appropriate*  for  all  programs  (as  we  also  note 
in  chaiiters  2  and  4).  But  much  of  the  potential  advantage  of  tcHThnical 
risk  as.sessment  is  lost  if  managers  and  reviewers  cannot  comiiare 
assessment  prcKCdures  and  results  in  general  terms  across  at  ic’ast  some 
programs,  such  as  those*  that  use  similar  technologies.  Comparability 
across  programs  facilitates  an  analysis  of  the  trad(*-offs  between  two  or 
more  systems  eomjic'ting  for  further  funding.  It  also  helps  reviewers  for- 
nuilate  and  follow  up  on  their  own  eoneerns  regarding  systems  with  sim¬ 
ilar  technical  features.  And,  finally,  eomparabiiiiy  across  prograncs 
reduet*s  the  time  it  take's  to  become  familiar  with  any  one  system  under 
review;  decisionmakers  do  not  have  to  learn  a  new  language  of  risk  (con¬ 
cepts,  procedures,  results)  for  each  system  they  examine.  We  reiterate 
our  resiHiiiM*  to  ikip’s  comments  under  "Finding  C":  tlie  task  is  to  find 
the  common  giound  m  definiiig  and  assessing  t('<  hnieal  risk,  is  Air  Force 
Regulation  7(M;")  doc*s  in  lis  definition  of  jirogram  isk. 


Recommendation  1  I'l  die  draft  that  i«>n  reviewi*!!,  we  inadi*  thri*e  recommendations  to  the 

Congress  and  four  to  thi*  ‘w'cretarj’  of  Ik-fense  For  the  final  draft,  we 
*  directed  all  the  reiomnu'iidations  to  the  secretary,  siibsiimiiig  the  topic 

of  contrar’tors’  risk  efforts  (originally  reeomnv'nd.ition  4  to  the  secre¬ 
tary)  under  what  is  nmv  reeemmeiidation  n.  The  lontent  of  .ill  the  rec¬ 
ommendations  is  the  same  in  the  puhiislu'fl  reisirt  as  in  the  draft  imi) 
reviewed. 
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Appt*n4ix  III 

Advaiicr  <  V^minentJi  fmm  tlie  l*>' 
DepATtnirni  of  iW'friUM* 


CoRnizant  of  difforena's  across  rx)r)  proRranis,  wo  supjKjrt  Riiidolincs 
that  arc  as  prociso  as  i)ossit)lo  and  as  floxiblc  as  necossar>'.  Tho  dovolop- 
mont  of  sucl.  guidoHnos  may  not  bo  oasy,  but  it  is  nonotholoss  critical  to 
cff'‘'‘t:v"  jirooram  I'nanagomt'nt  and  rcviow.  Sinco  it  is  often  iu‘ci“ssar>'  to 
ostiniato  tho  lik»*ii/.;>od  of  twhnical  problems  under  spc‘cilic  cost  or 
schedule  constraints,  definitions  and  prcK't'dures  should  b<‘  devisc-d  for 
each  comjKvnent  of  program  ris»<,  not  just  for  program  risk  in  genc*ral.  If 
an  estimate  of  technical  risk  is  requiri>d,  analysis  can  use  those  defini¬ 
tions  and  proerdures  to  provide  it. 


Recommendation  2  have  called  for  explu  u  attentio:i  to  ti*chniral  risk,  not  for  exclusive 

attention.  Technical  problems  should  be  describtxl  and  evaluated  clearly, 
not  left  implicit  in  assc^ssments  of  cost  or  schedule  risk.  As  we  have 
stall'd  in  the  reixirt,  we  support  risk  asw'ssment  at  least  twice — early 
and  late — in  each  acquisition  phav'.  to  inform  dc'cisionmakers  regarding 
work  in  that  phase  and  progress  to  the  next.  But  icchnical  risks  are 
ongoing  and,  as  ikid  U'lieves,  decisions  regarding  the  frt'quency  and  type 
of  assessment  are  liesl  left  to  program  man-'gers.  The  wording  of  this 
recommendation  was  changed  in  the  final  draft  in  order  to  <  larify  our 
position:  the  rii-ommendation  for  two  assessments  in  each  pha.se*  now 
calls  for  at  least  two  a.ssessmenLs  in  each  phase. 


Recommendation  3  purpose*  of  diK-umernation  is  to  make  it  possible  to  track  risks 

throughout  the  acquisition  cycle.  Only  if  records  are  kept  can  reviewers 
and  program  staff  fully  und<*rstand.  evaluate,  and  update  pa.st  avsess- 
menls.  The  records  need  not  be  lengthy,  so  long  as  they  adi.*quaiely 
describe  avsessment  pr(:cc*dures  and  results. 


Recommendation  4  hojK*  that  not/s  forthcoming  handbiHik  will  Ik*  detail<*d  enough  to 

provide  iisi  ful  guidance  regarding  format,  scoix*.  data  colleilion,  and 
a.ssi*ssment  approaches.  We  have  reporti*d  that  we  found  a  wide  \  anety 
of  risk  concepts  and  proc«*dure.s  among  the  2.)  program  offices  in  tlu* 
study.  We  also  found  general  inattention  to  the  complexity  and  maturity 
of  systems  as  asses.smenl  opiit'P.s  were  selected.  For  tht*se  rea.'^ms,  it  is 
im[K>rtant  that  fxipnot  merely  enumerate  various  risk  concepts  and  pn- 
ct'diiral  options  but  also  h  Tuiilale  advict*  for  st'lecting  apjiropriale  con¬ 
cepts  and  options. 
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Appendix  lU 

Advance  (  4kmnienu  from  the  t*>i. 
Depaitment  of  liefenae 


Recommendation  5  ood  already  n*quircs  risk  information  in  proj^ram  reviews,  but  the  infor¬ 

mation  now  provided  is  inadt'quatc  in  several  respects.  We  have  charac- 
terizi*d  the  kind  of  information  that  would  provide  an  adequate  basis  for 
understanding  and  evaluating  risk  assessment  procedures  and  that 
would  therefore  be  apimopriate  f^>r  inclusion  in  review  document.^.  For 
some  programs,  reviewers  may  already  know  the  technical  risks  or  may 
decide  that  they  do  not  need  all  the  information  we  would  make  avail¬ 
able.  Hut  it  is  important  that  the  information  be  available  for  every 
reviewer  who  wisht*s  to  see  it.  (This  position  underscores  the  need  for 
documenting  as.sessments.  Without  a  written  record,  information 
requested  later  by  reviewers  may  not  be  retrievable.) 


Recommendation  6  "e  btdieve  that  attention  to  technical  risk  should  be  explicit  rather  than 

being  left  implicit  in  cost  or  schedule  ri.sk  asse.vsmenLs.  We  agree  th,^t 
cost  and  schedule  nsks  also  require  canTu!  attention.  Hut  in  our  review, 
we  found  serious  inadixpiacies  in  uop’s  current  training  for  technical 
risk  as.sessmcnt.  Courses,  handbcxiks,  and  other  training  resources  rray 
require  formal  revision  to  ensure  full  and  proper  attention  to  risks  that 
are  di.stinctly  technical. 

Further,  as  we  have  noted  in  our  response  to  dod's  comment  under 
“Finding  G,"  it  is  imjiorttuit  to  distinguish  risk  as.sessment  from  risk 
management,  pod's  training  stres.ses  risk  management.  Hence,  wc  pro¬ 
pose  that  training  emphasize  not  just  technical  risk  but  also  ass<‘.s,sment. 
as  distinct  fn  m  management. 


Recommendation  4  (Now  in  J>«ve  not  proiK»stsi  that  technical  risk  assessments  be  required  of  ail 
Recommendation  5)  contractors.  Hut  when  a  contract  or  requi*st  for  proposal  does  make  the 

requirement,  specific  information  is  es.sential — the  same  information  we 
have  r(s-ommendi*d  for  program  offices'  documentation  of  risk.  The 
information  covers  format  and  scope  of  the  risk  ratings,  information 
sourcts,  data  colk“  tion,  and  the  analytic  approach. 
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